input: How does the book handle the topic of numerical methods for solving matrix problems?

answers: The book handles numerical methods by discussing various techniques such as the method of elimination, factorization by triangular matrices, and iterative methods for solving large systems of linear equations, emphasizing the importance of reducing rounding errors.

context:
# MATRIX THEORY

Joel N. Franklin

_Professor of Applied Mathematics
California Institute of Technology_

DOVER PUBLICATIONS, INC.

Mineola, New York
_Copyright_

Copyright © 1968, 1993 by Joel N. Franklin
All rights reserved.

_Bibliographical Note_

This Dover edition of _Matrix Theory,_ first published in 2000, is a slightly corrected, unabridged republication of the book originally published under the same title in 1968 by Prentice-Hall, Inc., Englewood Cliffs, New Jersey.

_Library of Congress Cataloging-in-Publication Data_

Franklin, Joel N.

Matrix theory / Joel N. Franklin.

p. cm.

Originally published: Englewood Cliffs, N.J. : Prentice-Hall, 1968, in series: Prentice-Hall series in applied mathematics.

Includes index.

ISBN-13: 978-0-486-41179-8

ISBN-10: 0-486-41179-6

1. Matrices. I. Title.

QA188.F66 2000
512.9'434—dc21

99-058316

Manufactured in the United States by Courier Corporation
41179603
www.doverpublications.com

## **PREFACE**

The widespread applicability of high-speed digital computers has made it necessary for every modern engineer, mathematician, or scientist to have a knowledge of matrix theory. The connection between digital computation and matrices is almost obvious. Matrices represent _linear_ transformations from a _finite_ set of numbers to another finite set of numbers. Since many important problems are _linear,_ and since digital computers with a finite memory manipulate only _finite_ sets of numbers, the solution of linear problems by digital computation usually involves matrices.

This book developed from a course on matrix theory which I have given at Caltech since 1957. The course has been attended by graduate students, seniors, and juniors majoring in mathematics, economics, science, or engineering. The course was originally designed to be a preparation for courses in numerical analysis; but as the attendance increased through the years, I modified the syllabus to make it as useful as possible for the many different purposes of the students. In many fields—mathematical economics, quantum physics, geophysics, electrical network synthesis, crystallography, and structural engineering, to name a few—it has become increasingly popular to formulate and to solve problems in terms of matrices.

Ten years ago there were few texts on matrices; now there are many texts, with different points of view. This text is meant to meet many different needs. Because the book is mathematically rigorous, it can be used by students of pure and applied mathematics. Because it is oriented towards applications, it can be used by students of engineering, science, and the social sciences. Because it contains the basic preparation in matrix theory required for numerical analysis, it can be used by students whose main interest is their future use of computers.

The book begins with a concise presentation of the theory of determinants. There follows a presentation of classical linear algebra, and then there is an optional chapter on the use of matrices to solve systems of linear differential equations. Next is a presentation of the most commonly used diagonalizations or triangularizations of Hermitian and non-Hermitian matrices. The following chapter presents a proof of the difficult and important matrix theorem of Jordan. Then there is a chapter on the variational principles and perturbation theory of matrices, which are used in applications and in numerical analysis. The book ends with a long chapter on matrix numerical analysis. This last chapter is an introduction to the subject of linear computations, which is discussed in depth in the advanced treatises of Householder, Varga, Wilkinson, and others.

The book presents certain topics which are relatively new in basic texts on matrix theory. There are sections on vector and matrix norms, on the condition-number of a matrix, on positive and irreducible matrices, on the numerical identification of stable matrices, and on the _QR_ method for computing eigenvalues.

A course on matrix theory lasting between one and two academic quarters could be based on selections from the first six chapters. A full-year course could cover the entire book.

The book assumes very little mathematical preparation. Except for the single section on the continuous dependence of eigenvalues on matrices, the book assumes only a knowledge of elementary algebra and calculus. The book begins with the most elementary results about determinants, and it proceeds gradually to cover the basic preparation in matrix theory which is necessary for every modern mathematician, engineer, or scientist.

I wish to thank Dr. George Forsythe and Dr. Richard Dean for reading parts of the manuscript and for suggesting the inclusion of certain special topics.

JOEL N. FRANKLIN

_Pasadena, California_

## **CONTENTS**

**1 Determinants**

1.1 Introduction

1.2 The Definition of a Determinant

1.3 Properties of Determinants

1.4 Row and Column Expansions

1.5 Vectors and Matrices

1.6 The Inverse Matrix

1.7 The Determinant of a Matrix Product

1.8 The Derivative of a Determinant

**2 The Theory of Linear Equations**

2.1 Introduction

2.2 Linear Vector Spaces

2.3 Basis and Dimension

2.4 Solvability of Homogeneous Equations

2.5 Evaluation of Rank by Determinants

2.6 The General _m_ × _n_ Inhomogeneous System

2.7 Least-Squares Solution of Unsolvable Systems

**3 Matrix Analysis of Differential Equations**

3.1 Introduction

3.2 Systems of Linear Differential Equations

3.3 Reduction to the Homogeneous System

3.4 Solution by the Exponential Matrix

3.5 Solution by Eigenvalues and Eigenvectors

**4 Eigenvalues, Eigenvectors, and Canonical Forms**

4.1 Matrices with Distinct Eigenvalues

4.2 The Canonical Diagonal Form

4.3 The Trace and Other Invariants

4.4 Unitary Matrices

4.5 The Gram-Schmidt Orthogonalization Process

4.6 Principal Axes of Ellipsoids

4.7 Hermitian Matrices

4.8 Mass-spring Systems; Positive Definiteness; Simultaneous Diagonalization

4.9 Unitary Triangularization

4.10 Normal Matrices

**5 The Jordan Canonical Form**

5.1 Introduction

5.2 Principal Vectors

5.3 Proof of Jordan's Theorem

**6 Variational Principles and Perturbation Theory**

6.1 Introduction

6.2 The Rayleigh Principle

6.3 The Courant Minimax Thorem

6.4 The Inclusion Principle

6.5 A Determinant-criterion for Positive Definiteness

6.6 Determinants as Volumes; Hadamard's Inequality

6.7 Weyl's Inequalities

6.8 Gershgorin's Theorem

6.9 Vector Norms and the Related Matrix Norms

6.10 The Condition-Number of a Matrix

6.11 Positive and Irreducible Matrices

6.12 Perturbations of the Spectrum

6.13 Continuous Dependence of Eigenvalues on Matrices

**7 Numerical Methods**

7.1 Introduction

7.2 The Method of Elimination

7.3 Factorization by Triangular Matrices

7.4 Direct Solution of Large Systems of Linear Equations

7.5 Reduction of Rounding Error

7.6 The Gauss-Seidel and Other Iterative Methods

7.7 Computation of Eigenvectors from Known Eigenvalues

7.8 Numerical Instability of the Jordan Canonical Form

7.9 The Method of Iteration for Dominant Eigenvalues

7.10 Reduction to Obtain the Smaller Eigenvalues

7.11 Eigenvalues and Eigenvectors of Tridiagonal and Hessenberg Matrices

7.12 The Method of Householder and Bauer

7.13 Numerical Identification of Stable Matrices

7.14 Accurate Unitary Reduction to Triangular Form

7.15 The _QR_ Method for Computing Eigenvalues

**INDEX**

## **NOTATION USED IN THIS BOOK**

Some authors denote vectors by boldface ( **x** , **y** , **z** ), but we shall simply denote them by using lower-case English letters ( _x_ , _y_ , _z_ ). If we wish to designate the components of a vector, _x_ , we shall use subscripts; thus, _x_ 1, . . . , _x_ _n_ designates the components of _x_. Superscripts will be used to designate different vectors; thus, _x_ 1, _x_ 2, _x_ 3 designates three vectors, and if there appears to be any chance of confusion with the powers of a scalar, we shall enclose the superscripts in parentheses—e.g., _x_ (1), _x_ (2), _x_ (3). Thus,  , . . . ,   designates the _n_ components of the vector _x_ (2), whereas  , . . . ,   designates the squares of the _n_ components of a vector _x_.

Matrices will be denoted by capital letters ( _A_ , _B_ , _C_ ). Subscripts may be used to designate different matrices—e.g., _A_ 1, _A_ 2, _A_ 3, . . . —but _A_ , _A_ 2, _A_ 3 will designate different powers of the same square matrix, _A_. Thus,

A lower case Greek letter—e.g., λ, _α_ , _γ_ —will always designate an ordinary real or complex number. A Greek letter will never be used to designate a vector or a matrix. We will also occasionally use subscripted English letters—e.g., _c_ 1, . . . , _c_ _n_ —to designate real or complex numbers. Subscripted English letters will _never_ be used to designate vectors; thus, _z_ 3 cannot designate a vector, although it may be used to designate the third component of a vector _z._

For the columns of a matrix A we will often use the notation _a_ (1), . . . , _a_ ( _n_ ) or simply _a_ 1, . . . , _a_ _n_. For the components of a matrix _A_ we will write _a_ _ij_. Sometimes we shall write

to indicate that _A_ is the matrix

If _A_ has the same number, _n_ , of rows and columns, we may write

The rows of a matrix are horizontal; the columns are vertical. Thus,

may be the third row of a matrix, whereas

may be the seventh column. Since a column vector takes much space to print, we will sometimes refer to it by the prefix "col." Thus, the preceding vector may be written as col ( _a_ 17, . . . , _a_ _m_ 7).

The superscripted letter _e_ has a particular meaning throughout the book. The vector _e_ _j_ is a column-vector with its _j_ th component equal to 1 and with all other components equal to 0. Thus, if there are five components,

## **1 DETERMINANTS**

### **1.1 INTRODUCTION**

Suppose that we wish to solve the equations

To solve for _x_ 1 we multiply the first equation by 8, multiply the second equation by 7, and subtract the resulting equations. This procedure gives

Thus, _x_ 1 = −3/−5 =  . To find _x_ 2, multiply the second equation by 2, the first equation by 3, and subtract. The result is

Thus _x_ 2 = −2/−5 =  . To generalize these equations, write

In the example (1)

Solving for _x_ 1 and _x_ 2, we find

If we introduce the notation

formula (2) yields

if the denominator in each fraction is not zero.

_A rectangular collection of numbers is a matrix_. Thus,

are matrices. So is the number 17—a single number is a 1 × 1 matrix. Systems of linear equations are completely specified by the matrices containing their coefficients and by their right-hand sides. _A vector is a matrix with only one column or only one row_. Thus

are vectors. The first is a _row vector_ ; the second, a _column vector_.

A determinant is a single number computed from a square matrix. We speak of "the determinant of a matrix," and in the next section, we shall define the determinant of an _n_ × _n_ matrix. For _n_ = 2 we define

For example, the collection of four coefficients

from equations (1) is a _matrix_ with a _determinant_ that is the single number

As we saw in (4), determinants appear in the solution of linear equations.

### **1.2 THE DEFINITION OF A DETERMINANT**

We wish to generalize our solution of two equations in two unknowns to _n_ equations in _n_ unknowns. For _n_ = 3 write

If _x_ 2 and _x_ 3 are eliminated, we can obtain by a long computation the formula

where

and where ∆1 is found by substituting _b_ 1, _b_ 2, _b_ 3, respectively, for _a_ 11, _a_ 21, _a_ 31 in the expression for ∆. A second enormous computation would yield

where ∆2 is found by substituting _b_ 1, _b_ 2, _b_ 3, respectively, for _a_ 12, _a_ 22, _a_ 32 in the expression for ∆. A third computation would give

where ∆3 is found by substituting _b_ 1, _b_ 2, _b_ 3, respectively, for _a_ 13, _a_ 23, _a_ 33. In a later section these results will be proved and generalized. Here we only wish to motivate the definition of the determinant.

Every term in the expression for ∆ has the form _a_ 1 _j_ _a_ 2 _k_ _a_ 3 _l_ where _j_ , _k_ , _l_ is a permutation of 1, 2, 3. With each term there is a sign ±1 = _s_ ( _j_ , _k_ , _l_ ), which we call the _sign of the permutation_ j, k, l. Thus

Evidently,

Now (3) takes the form

where the summation extends over all six permutations _j_ , _k_ , _l_.

For an _n_ × _n_ matrix ( _a_ _ij_ )( _i_ , _j_ = 1, . . . , _n_ ) we define the determinant ∆ as the sum of _n_! terms

The summation extends over all _n_! permutations _j_ 1, . . . , _j_ _n_ of 1, . . . , _n_. The sign _s_ ( _j_ 1, . . . , _j_ _n_ ) is defined as

In other words, _s_ = 1 if the product of all _j_ _q_ – _j_ _p_ for _q_ > _p_ is positive; _s_ = −1 if the product is negative.

For example, the determinant of a 5 × 5 matrix has 120 terms. One of the terms is – _a_ 13 _a_ 25 _a_ 31 _a_ 42 _a_ 54. The minus sign appears because

It should be emphasized that so far we have proved nothing. We merely have a definition (9) which we hope will be useful. For _n_ = 2, the definition gives

which is consistent with the definition given in the preceding section. For _n_ = 1, we define ∆ = _a_ 11, _s_ (1) = 1.

#### **PROBLEMS**

**1.** Verify formula (7) for the six permutations in formula (6).

**2.** In the expansion (9) of the determinant of a 7 × 7 matrix there will be a term ± _a_ l7 _a_ 26 _a_ 35 _a_ 44 _a_ 53 _a_ 62 _a_ 71. Is the sign plus or is it minus?

**3.** Consider the equations

Evaluate the determinants ∆, ∆1, ∆2, ∆3 and solve for _x_ 1, _x_ 2, _x_ 3 by formulas (2), (4), and (5).

### **1.3 PROPERTIES OF DETERMINANTS**

The definitions (9) and (10) in the last section show that, to study determinants, we must study permutations.

**Theorem 1.** _If two numbers in the permutation_ j1, . . . , jn _are interchanged_ , _the sign of the permutation is reversed_. For example,

_Proof_. Suppose that the two numbers are adjacent, say _k_ and _l_ , in the permutation _j_ 1, . . . , _k_ , _l_ , . . . , _j_ _n_. When _k_ and _l_ are interchanged, the product ∏ ( _j_ _s_ – _j_ _r_ ) for _s_ > _r_ is unchanged except that the single term _l_ – _k_ becomes _k_ – _l_. Therefore, the sign is reversed.

If _k_ and _l_ are not adjacent, let them be separated by _m_ numbers. Move _k_ to the right by _m_ successive interchanges of adjacent numbers so that _k_ is just to the left of _l_. Now move _l_ to the left by _m_ \+ 1 successive interchanges of adjacent numbers. The effect of these interchanges is simply to interchange _k_ and _l_ in the original permutation. Since the sign is reversed an odd number of times, namely _m_ \+ _m_ \+ 1 times, the sign is reversed when _k_ and _l_ are interchanged.

**Theorem 2.** _Let the permutation_ j1, . . . , jn _be formed from_ 1, 2, . . . , n _by_ t _successive interchanges of pairs of numbers_. _Then_ s(j1, . . . , jn) = (−1)t.

_Proof_. According to the last theorem, the sign of the permutation 1, 2, . . . , _n_ is reversed _t_ times as the permutation _j_ 1, . . . , _j_ _n_ is formed. The result follows because _s_ (l, 2, . . . , _n_ ) = 1. For example,

If _t_ is even, the permutation _j_ l, . . . , _j_ _n_ is called an _even permutation_ ; if _t_ is odd, _j_ is called an _odd permutation_. Since the sign of a permutation is uniquely defined, a permutation cannot be both even and odd. _The number of even permutations equals the number of odd permutations_ = _n_!/2 if _n_ > 1, since the even permutations _j_ 1, _j_ 2, . . . , _j_ _n_ may be put into a one-to-one correspondence with the odd permutations by the single interchange of _j_ 1 and _j_ 2.

The _transpose of a matrix_ is formed by interchanging corresponding rows and columns. Thus, if _A_ = ( _a_ _ij_ ), the component _b_ _ij_ of _A_ _T_ equals _a_ _ji_. For example,

Another example is

**Theorem 3.** _If A is a square matrix_ , det A = det AT.

_Proof_. Let _B_ = ( _b_ _ij_ ) = _A_ _T_ = ( _a_ _ji_ ). By definition

where the summation extends over all _n_! permutations, _j_ = _j_ 1, . . . , _j_ _n_ Since _b_ _ij_ = _a_ _ji_ ,

By rearranging the terms, we may write

For example, _a_ 31 _a_ 12 _a_ 23 = _a_ 12 _a_ 23 _a_ 31. The rearrangement (2) establishes a one-to-one correspondence between permutations _j_ and permutations _i_. Therefore, as _j_ goes through all _n_! permutations in the summation (1), _i_ goes through all permutations. If a rearrangement from the left-hand side of (2) can be accomplished by _t_ interchanges of pairs of _a_ 's, then the right-hand side can be rearranged by _t_ interchanges to produce the left-hand side. Thus,

implies that

Therefore, _s_ ( _j_ ) = (−1) _t_ = _s_ ( _i_ ), and

**Theorem 4.** _If two rows of a square matrix_ A _are interchanged_ , _the sign of the determinant is reversed_. _Or_ , _if two columns are interchanged_ , _the sign is reversed_. For example,

_Proof_. Suppose that rows _r_ and _s_ are interchanged in the matrix _A_ to produce a matrix _B_. Then

By definition,

Let _k_ be the permutation produced from _j_ by interchanging _j_ _r_ and _j_ _s_. This interchange establishes a one-to-one correspondence between permutations _j_ and _k_. The sign _s_ ( _k_ ) = – _s_ ( _j_ ). Interchanging _b_ _rj_ _r_ with _b_ _sj_ _s_ gives

Therefore,

To establish the column-analog, we use Theorem 4. Let _C_ be produced by interchanging two columns in _A_. Then _C_ _T_ is produced by interchanging two rows in _A_ _T_. Therefore,

**Corollary.** _If two rows_ , _or two columns_ , _of a square matrix are identical_ , _the determinant is zero_.

_Proof_. If the two identical rows or columns are interchanged, the matrix is unaltered, but the determinant must change sign. Therefore, the determinant equals minus itself, and is thus equal to zero.

**Theorem 5.** _If a row or a column of a square matrix is multiplied by a constant_ , c, _the determinant is multiplied by_ c. For example,

_Proof_. Each term _a_ 1 _j_ 1 . . . _a_ _nj_ _n_ in the expansion of det _A_ contains exactly one component from each row. Therefore, if any one row is multiplied by _c_ , the determinant is multiplied by _c_. Each term _a_ l _j_ 1 . . . _a_ _nj_ _n_ contains exactly one term from each column. Therefore, if a column is multiplied by _c_ , the determinant is multiplied by _c_.

**Theorem 6.** _If a multiple of one row_ ( _column_ ) _is subtracted from another row_ ( _column_ ) _of a matrix_ , _the determinant is unchanged_. For example,

_Proof_. By the transpose theorem, Theorem 3, we only need to prove the row part of Theorem 6. Let λ times row _r_ be subtracted from row _s_. Then _a_ _sj_ _s_ is replaced by _a_ _sj_ _s_ – λ _a_ _rj_ _s_. For definiteness, assume that _r_ < _s_. The new matrix has the determinant

The first sum equals det _A_. The second sum is zero because it is the expansion of a determinant with identical rows _r_ and _s_. This completes the proof.

Theorem 6 is used to evaluate determinants by reducing them to triangular form. For example,

The last determinant equals 1 because of the following result:

**Theorem 7.** _If_ a1j = 0 _for_ i > j, _then_ det A = a11a22 . . . ann.

_Proof_. A term _s_ ( _j_ ) _a_ l _j_ 1 . . . _a_ _nj_ _n_ in the expansion of det _A_ can be nonzero only if  . But the only permutation _j_ that satisfies all of these inequalities is _j_ = 1, 2, . . . , _n_. Therefore, det _A_ equals the single term _a_ 11 _a_ 22 . . . _a_ _nn_.

**Theorem 8.** aij = 0 _for_ i < j, _then_ det A = a11a22 . . . ann.

_Proof_. det _A_ = det _A_ _T_ = _a_ 11 _a_ 22 . . . _a_ _nn_.

#### **PROBLEMS**

**1.** By applying Theorems 4 and , show that

**2.** Let _A_ have the form

Show that

Use an argument similar to the proof of Theorem 7.

**3.** Generalize the result of Problem 2. Let _A_ be a _block_ -triangular matrix

Show that

Can you prove this result by induction from the result of Problem 2?

**4.** Is it possible to rearrange the letters of the alphabet _a_ , _b_ , . . . , _z_ in reverse order _z_ , _y_ , . . . , _a_ by exactly 100 successive interchanges of pairs of letters?

### **1.4 ROW AND COLUMN EXPANSIONS**

If _A_ is a square matrix, we may regard det _A_ as a function of the elements _a_ 11, _a_ 12, . . . , _a_ 1 _n_ in the first row. Each term _a_ l _j_ 1 _a_ 2 _j_ 2 . . . _a_ _nj_ _n_ contains exactly one element, _a_ 1 _j_ 1, from the first row. Therefore, we may write the determinant as a linear combination

From the definition of det _A_ we observe that

where the summation ∑ _k_ is extended over all ( _n_ – 1)! permutations _j_ = _j_ 1, _j_ 2, . . . , _j_ _n_ with _j_ 1 = _k_. Note that

where

since the numbers 1, 2, . . . , _k_ – 1 appear to the right of _k_ in the permutation _j_ = _k_ , _j_ 2, . . . , _j_ _n_. Therefore,

where the summation extends over all the ( _n_ – 1)! permutations _j_ 2, . . . , _j_ _n_ of the _n_ – 1 numbers 1, . . . , _k_ – 1, _k_ \+ 1, . . . , _n_. Thus,

where _A_ l _k_ is the ( _n_ – 1) × ( _n_ – 1) matrix formed by eliminating row 1 and column _k_ from _A_.

Formulas (1) and (5) give the _expansion of_ det A _by the first row_. For example,

Suppose that we wish to expand the determinant (6) by the third row. We bring the third row to the top, without disturbing the order of the other rows, by interchanging rows 3 and 2 and then interchanging rows 2 and 1.

We now expand by the first row to obtain

This illustrates the general result:

**Theorem 1.** _Let_ A = ( _a_ ij) _be an_ n × n _matrix_ , _where_  . _Let_ Aij _be the_ (n − 1) × (n − 1) _matrix formed by deleting row_ i _and column_ j _from_ A. _Defining the cofactors_

_we then have the expansion by row_ i:

_and we have the expansion by column_ j:

_Proof_. We have already proved (9) for _i_ = 1 in formulas (1) and (5). For _i_ > 1 bring row _i_ to the top by _i_ − 1 successive interchanges of pairs of rows: _i_ and _i_ − 1, _i_ − 1 and _i_ − 2, . . . , 2 and 1. Call the resulting matrix _B_. Because of the _i_ − 1 interchanges,

Expand det _B_ by its first row.

Since _b_ 1 _j_ = _a_ _ij_ and _B_ 1 _j_ = _A_ _ij_ , we have

Therefore, by (12),

This establishes the row expansion (10).

To establish the column expansion (11), let _R_ = _A_ _T_. Expand _R_ by row _j_ :

But _r_ jk = _a_ kj and  . For example, if _n_ = 3,

Therefore, since det _A_ = det _A_ _T_ ,

This is the expansion by column _j_.

As an example of a row expansion, we shall evaluate _Vandermonde's determinant_ :

If _x_ _i_ = _x_ _j_ for some _i_ ≠ _j_ , the determinant equals 0. Suppose that _x_ 1, . . . , _x_ _n_ are distinct. We regard _V_ _n_ as a function of the variable _x_ ≡ _x_ _n_ , and we regard _x_ 1, . . . , _x_ _n_ −1 as constants. Now _V_ _n_ is a polynomial of degree _n_ − 1 in _x_ , and _V_ _n_ has the distinct roots _x_ = _x_ 1, _x_ = _x_ 2, . . . , _x_ = _x_ _n_ −1. Therefore,

where _α_ is independent of _x_. But _α_ is the coefficient of  . Expanding the determinant (14) by its last row, we see that _α_ is the cofactor belonging to  . But this cofactor is the lower-order Vandermonde determinant,

Direct computation for _n_ = 2 gives

Now (15) gives

We conjecture

But if

then (15) gives

which establishes (17) by induction. The identity (17) is correct even if the _x_ _i_ are not distinct because in that case both sides are zero.

#### **PROBLEMS**

**1.** Evaluate

by the three row expansions and by the three column expansions.

**2.** By certain simplifications, evaluate

**3.** Give a simple expression for

**4.** Give a simple expression for

Note that this determinant, as a function of _a_ , is a quartic polynomial with roots _b_ , _c_ , and _d_ ; and the coefficient of _a_ 3 is zero.

### **1.5 VECTORS AND MATRICES**

A set of _m_ linear equations in _n_ unknowns

may be represented in a compact form

Here _A_ is the _matrix_

and _x_ and _y_ are the _column vectors_

If _A_ is fixed while _x_ is allowed to vary over all column vectors with _n_ components, the equation _Ax_ = _y_ gives a _mapping_ from the "space" _X_ of all _n_ -component vectors _x_ into the "space" _Y_ of all _m_ -component vectors _y_. When we solve the linear equations (1) we find those "points" _x_ in _X_ which are mapped into a given "point" _y_ in _Y_ by the matrix _A_.

A mapping _Ax_ = _y_ might be followed by another mapping, _By_ = _z_ , from _Y_ into the space _Z_ of _p_ -component column-vectors _z_. Let _B_ be the matrix

The successive mappings _Ax_ = _y_ , _By_ = _z_ map _X_ into _Z_. Explicitly,

yield

or, equivalently,

Therefore, the successive mappings _Ax_ = _y_ , _By_ = _z_ yield a mapping _Cx_ = _z_ , where _C_ is the _p_ × _n_ matrix

In this case we write

_The product_ BA _of the matrices_ B _and_ A _is defined as the matrix_ C _with the components_ (6).

_Two matrices are called equal if their corresponding components are equal_. As a rule, _BA_ ≠ _AB_. The product is not even defined unless the number of columns of the matrix on the left equals the number of rows of the matrix on the right. Thus, if

we have

but _AB_ is undefined. If

we have

The inequality _AB_ ≠ _BA_ means that the transformation _AB_ coming from _B_ followed by _A_ , is different from the transformation _BA_ coming from _A_ followed by _B_.

If _Ax_ = _y_ , _By_ = _z_ , and _Dz_ = _w_ we may write

or

Thus, the product of three matrices may be taken in either order:

The _i_ , _j_ component of _DBA_ is the double sum

The inequality _AB_ ≠ _BA_ states that _matrix multiplication is not commutative_ , The equality ( _DB_ ) _A_ = _D_ ( _BA_ ) states that _matrix multiplication is associative_.

Sometimes it is convenient to multiply every component of a vector _x_ or a matrix _A_ by the same number _α_. We then write

Obviously, multiplication of a matrix or vector by a number is commutative. Thus, _αABx_ = _A_ ( _αB_ ) _x_ = _AB_ ( _αx_ ).

#### **PROBLEMS**

**1.** Let _A_ and _B_ be the matrices

Form the products _AB_ and _BA_.

**2.** Let

and let

Express _z_ 1 and _z_ 2 in terms of _x_ 1 and _x_ 2. If _y_ = _Ax_ and _z_ = _By_ , what is the product _BA_?

**3.** If _y_ = _Ax_ for all _x_ , and if _z_ = _By_ for all _y_ , we have shown that _z_ = _Cx_ for all _x_ , where _C_ is defined in (6). Could there be another matrix, _C_ ′, with this property? Suppose that _z_ = _Cx_ = _C_ ′ _x_ for _all x_ ; show that _C_ = _C_ ′. [Hint: Let _x_ , successively, be the unit vectors column (1,0, . . . , 0), column (0, 1, 0, . . . ), . . . , column (0, 0, . . . , 1).]

**4.** If _y_ = _Ax_ can be solved for _x_ in terms of _y_ by a formula _x_ = _A_ ′ _y_ , we call _A_ ′ an _inverse_ of _A_ , and we write _A_ ′ = _A_ −1. Compute an inverse _A_ −1 for

**5.** If _A_ −1 is an inverse of _A_ , and if _B_ −1 is an inverse of _B_ , what is an inversefor _BA_ expressed as a matrix product?

**6.** Let _A_ be an _m_ × _n_ matrix. Let _y_ = _Ax_ for all _x_ Define the _row_ -vectors _x_ ′ = ( _x_ 1, . . . , _x_ _n_ ) and _y_ ′ − ( _y_ 1, . . . , _y_ _m_ ), where _x_ and _y_ are the column-vectors with the same components. How can you construct an _n_ × _m_ matrix, _A_ ′, such that _y_ ′ = _x_ ′ _A_ ′ for all _x_ ′? If

what is _A_ ′?

**7.** Define _A_ ′ as in the last problem. Express ( _AB_ )′ in terms of _A_ ′ and _B_ ′

### **1.6 THE INVERSE MATRIX**

Let a system of _n_ equations in _n_ unknowns be written in matrix-vector form : _Ax_ = _b_. _Define the_ n × n _identity matrix_

_as the matrix with ones on the main diagonal and zeros off the main diagonal_. We can solve _Ax_ = _b_ if we can find an _n_ × _n_ matrix _B_ such that _AB_ = _I_. A solution is _x_ = _Bb_ because _Ax_ = _A_ ( _Bb_ ) = ( _AB_ )b = _Ib_ = _b_.

**Theorem 1.** _Let_ A _be an_ n × n _matrix_ , _with_ n > 1. _Let_ Aij _be the_ (n − 1) × (n − 1) _matrix formed by deleting row_ i _and column_ j _from_ A. _Define the cofactor matrix_

_Let_ ∆ = det _A_. _Then_

_If_ ∆ ≠ 0, _then_ A _has an inverse matrix_ B = ∆−1CT _satisfying_

**E XAMPLE 1.** Let

Then

In this example, det _A_ = 0. We will show later that det _A_ = 0 implies that no matrix _B_ exists such that either _AB_ = _I_ or _BA_ = _I_.

**E XAMPLE 2.** Let

Then

Since ∆ ≠ 0, we have

_Proof ofTheorem 1_. Since (4) follows by dividing (3) by the number ∆, we only need to prove (3). Let _D_ = _AC_ _T_ , _E_ = _C_ _T_ _A_. We wish to show that

where

The symbol δ _ij_ defines the _Kronecker delta_ ; it is a useful abbreviation.

If _C_ = ( _c_ _ij_ ), then

But this is the sum (10) in Section 1.4 in the row-expansion theorem. Therefore, _d_ _ij_ = ∆ ( _i_ = 1, . . . , _n_ ). Similarly,

by the column expansion (11) in Section 1.4. This proves (5) for _i_ = _j_.

For _i_ ≠ _j_ , we have

If _a_ _ik_ were replaced by _a_ _jk_ in (7), the sum would be the expansion of det _A_ by row _j_. Let _A_ ′ be the matrix formed by replacing row _j_ of _A_ by row _i_. Thus, _A_ ′ has identical rows _i_ and _j_. Further,   because the cofactors of elements in row _j_ are unaffected by modifications of row _j_. Expanding det _A_ ′ by row _j_ , we find

But det _A_ ′ = 0 because _A_ ′ has two equal rows. Therefore, _d_ _ij_ = 0 if _i_ ≠ _j_. Similarly,

is the expansion by column _i_ of the determinant of the matrix _A_ ″ formed from _A_ by replacing column _i_ by column _j_. Since _A_ ″ has two equal columns, det _A_ ″ = _e_ _ij_ = 0. This completes the proof of the theorem.

**Theorem 2.** (Cramer's rule) _Let_ A = (aij) _be an_ n × n _matrix_. _Let_ ∆ = det A ≠ 0. _Then the equations_

_have the unique solution_

_where_ ∆j _is the determinant of the matrix formed by replacing the_ j _th column of_ A _by the vector_ b.

**E XAMPLE.** For _n_ = 2 this rule states

_Proof_. Let the cofactor matrix _C_ and the inverse matrix _B_ be defined as in Theorem 1. The matrix form of (8) is _Ax_ = _b_. Letting _x_ = _Bb_ , we see that _x_ = _Bb_ is a solution because _AB_ = _I_. Multiplying _Ax_ = _b_ on the left by _B_ , we see that _x_ = _Bb_ is the only possible solution because _BA_ = _I_.

The _j_ th component of _x_ = _Bb_ is

But the sum in (10) is just the expansion of ∆ _j_ by column _j_. Therefore, _x_ _j_ = ∆−1 ∆ _j_.

#### **PROBLEMS**

**1.** Define

Compute _A_ −1 and _B_ −1. Compute ( _A_ \+ _B_ )−1.

**2.** Solve

by multiplying the equation by the inverse matrix.

**3.** Solve the equation in the last problem by Cramer's rule.

**4.** Solve

by Cramer's rule.

**5.** If det _A_ ≠ 0, show that ( _A_ _T_ )−1 = ( _A_ −1)T.

**6.** * Let _A_ be an _n_ × _n_ matrix. Let _A_ 2 = _A_ · _A_ , _A_ 3 = _A_ · _A_ · _A_ , etc. Let λ be a complex number. Suppose that the series of matrices

converges, i.e., that each of the _n_ 2 components is a convergent series of complex numbers. Show that _S_ = ( _I_ − λ _A_ )−1. Find a series for ( _I_ λ _A_ )−2.

**7.** Let _A_ be an _n_ × _n triangular_ matrix, with _a_ _ij_ = 0 for _j_ > _i_. Assuming all _a_ _ii_ ≠ 0, prove that _A_ −1 is also a triangular matrix.

### **1.7 THE DETERMINANT OF A MATRIX PRODUCT**

To show that the inverse of _A_ exists and is unique _only_ if det _A_ ≠ 0 we shall use the following theorem:

**Theorem 1.** _Let_ A _and_ B _be_ n × n _matrices_. _Then_

**E XAMPLE.** Let

Then det _A_ = -2, det _B_ = -10, so (det _A_ )(det _B_ ) = 20. But

_Proof of the Theorem_. Let _P_ = _AB_. We will show that det _P_ = (det _A_ ) (det _B_ ). By definition,

But

Therefore,

But the last sum equals a determinant:

This determinant is zero if _k_ _α_ = _k_ _β_ for any _α_ ≠ _β_ , for then it is the determinant of a matrix with identical rows _α_ and _β_. In other words, the determinant (5) is zero unless _k_ 1, _k_ 2,..., _k_ _n_ = a permutation, _k_ , of the numbers 1, 2,..., _n_. Therefore, in the summation (4) we may eliminate all of the _n_ _n_ combinations _k_ 1,..., _k_ _n_ except the _n_! permutations _k_. Therefore,

The determinant (5) can be reduced to ± det _B_ if kl,..., _k_ _n_ is a permutation. Let _k_ be reduced to 1,..., _n_ by _t_ interchanges of pairs of numbers. If the corresponding pairs of rows are interchanged in (5), we find

But (–1) _t_ = _s_ ( _k_ ), the sign of the permutation _k_. Now (6) becomes

**Theorem 2.** _Let_ A _be an_ n × n _matrix_. _If det_ A = 0, _then for some_ b _the equation_ Ax = b _has no solution_ x. _Equivalently_ , _if the equation_ Ax = b _is solvable for every_ b, _then det_ A ≠ 0.

_Proof_. Let _e_ 1, _e_ 2,..., _e_ _n_ be the columns of _I_ :

If every equation _Ax_ = _b_ is solvable, then in particular the equations _Ax_ = _e_ 1,..., _Ax_ = _e_ _n_ have solutions _x_ = _x_ 1,..., _x_ = _x_ _n_. Let _X_ be the _n_ × _n_ matrix with columns _x_ 1,..., _x_ _n_ (each _x_ _j_ is an _n_ -component column vector). Then

Then, by Theorem 1,

Therefore, the factor det _A_ must be nonzero.

**Theorem 3.** _If_ det A = 0, _no matrix_ B _can be found for which either_ AB = I _or_ BA = I.

_Proof_. _I_ = _AB_ implies 1 = det ( _AB_ ) = (det _A_ )(det _B_ ), so that det _A_ ≠ 0. Similarly, _I_ = _BA_ implies det _A_ ≠ 0.

**Theorem 4.** _If det_ A = ∆ ≠ 0, _define_

_Then_ AB = I _if and only if_ B = A–1. _Similarly_ , BA = I _if and only if_ B = A–1. _Every equation_ Ax = b _has the unique solution_ × = A–1b.

_Proof_. In the last section we proved that _A_ –l _A_ = _AA_ –1 = _I_. Conversely, suppose _AB_ = _I_ for some matrix _B_. Multiply both sides by _A_ –l on the left; then

If _BA_ = _I_ , multiply both sides by _A_ –1 on the right; then

A square matrix _A_ is called _singular_ if det _A_ = 0; it is called _nonsingular_ if det _A_ ≠ 0.

#### **PROBLEMS**

**1.** For the matrices

verify the identity det ( _AB_ )= (det _A_ )(det _B_ ).

**2.** The function _ϕ_ ( _A_ ) = det _A_ satisfies the relation _ϕ_ ( _AB_ ) = _ϕ_ ( _BA_ ) even if _AB_ ≠ _BA_. Show that another function that satisfies this relation, is _ϕ_ ( _A_ ) = trace _A_ = ∑ _a_ _ii_.

**3.** Let _A_ be a 3 × 2 matrix. Show that there is some vector _b_ = col ( _β_ 1, _β_ 2, _β_ 3) for which the equation _Ax_ = _b_ is unsolvable. (Hint: Form a square matrix _A′_ by adjoining a column of zeros to _A_ , and note that the equation _Ax_ = _b_ is equivalent to an equation _A′x′_ = _b_ , where det _A′_ = 0.)

**4.** Generalize the preceding result. Let _A_ be an _m_ × _n_ matrix with _m_ > _n_. Show that some equation _Ax_ = _b_ is unsolvable.

**5.** Assume that _m_ > _n_. Let _A_ be an _m_ × _n_ matrix, and let _B_ be an _n_ × _m_ matrix. Show that det ( _AB_ ) =0 by proving that some equation _ABy_ = _b_ has no solution. Use the result of Problem 4.

**6.** * Assume that _m_ < _n_. Let _A_ be an _m_ × _n_ matrix, and let _B_ be an _n_ × _m_ matrix. Let _α_ ( _k_ 1, _k_ 2,..., _k_ m) be the determinant formed from _columns k_ 1, _k_ 2,..., _k_ _m_ of _A_. Let _β_ ( _k_ 1, _k_ 2,..., _k_ _m_ ) be the determinant formed from _rows k_ 1, _k_ 2,..., _k_ _m_ of _B_. Using the method in the proof of Theorem 1, show that

If _m_ = 2 and _n_ = 3, this identity becomes det ( _AB_ ) =

= _α_ (1,2) _β_ (1,2) + _α_ (1,3) _β_ (1,3) + _α_ (2,3) _β_ (2,3)

### **1.8 THE DERIVATIVE OF A DETERMINANT**

For certain problems in analysis we require the derivative ∆′(λ) of a determinant ∆(λ) = det _A_ (λ) = det [ _a_ _ij_ (λ)].

**Theorem 1.** _Let_ aij(λ) _be differentiable functions of_ λ(i, j = 1,..., n). _Then_

_where_ ∆(λ) = det [aij(λ)], _and where_ ∆k(λ) _is the determinant formed by replacing the_ k _th row_ akj(λ) (j = 1,..., n) _by the row of derivatives_   (j = l, . . ., n).

**E XAMPLE 1.**

_Proof of the Theorem_. By definition

From calculus we know that

Now (2) becomes

**E XAMPLE 2.** Let ∆(λ) = det (λ _I_ – _B_ ), where _B_ is an _n_ × _n_ matrix of constants ( _n_ > 1), for example,

Then

where _B_ 1 is formed from _B_ by deleting row 1 and column 1. Similarly, ∆ _k_ (λ) = det (λ _I_ – _B_ _k_ ), where _B_ _k_ is formed from _B_ by deleting row _k_ and column _k_. The identity (1) now states

#### **PROBLEMS**

**1.** Differentiate the determinant

**2.** Let _A_ (λ) be an _n_ × _n_ matrix. Let δ _k_ (λ) be the determinant of the matrix formed by replacing the _k_ th _column_ of _A_ by the column of derivatives.

Prove that

**3.** Let _ϕ_ 1(λ) and _ϕ_ 2(λ) be different solutions of the differential equation _ϕ_ ″(λ) + _a_ (λ) _ϕ_ ′(λ) + _b_ (λ) _ϕ_ (λ) = 0. Define the _Wronskian_

Prove that

**4.** Generalize the preceding result for linear differential equations of order _n_.

**5.** Let _A_ (λ) be a 2 × 2 matrix. Let _x_ (λ) be a vector satisfying the differential equation _x_ ′(λ) = _A_ (λ) _x_ (λ), i.e.,

Let _y_ (λ) also satisfy _y_ ′(λ) = _A_ (λ) _y_ (λ). Form the Wronskian

Prove that

**6.** Generalize the preceding result for the differential equation _x_ ′(λ) = _A_ (λ) _x_ (λ), where _A_ is an _n_ × _n_ matrix whose components are continuous functions of λ. Let _x_ (1)(λ),..., _x_ ( _n_ )(λ) be different solutions. Find a first-order differential equation for the Wronskian. By solving this differential equation, show that either the Wronskian is zero for all λ, or the Wronskian is never zero. This is an important result in the theory of differential equations.

## **2 THE THEORY OF LINEAR EQUATIONS**

### **2.1 INTRODUCTION**

In the last chapter we showed that a system _Ax_ = _b_ of _n_ equations in _n_ unknowns has one and only one solution _if_ det _A_ ≠ 0. If _b_ ≠ 0, the system is called _inhomogeneous_ ; if _b_ = 0, the system is called _homogeneous_.

We have not discussed systems with more equations than unknowns.
We have not discussed systems with more unknowns than equations.

Most importantly, we have not discussed homogeneous systems _Ax_ = 0 where _A_ is an _n_ × _n_ matrix with det _A_ = 0. As we shall see in Chapter 3, _homogeneous linear systems of equations characterize the solutions of linear systems of differential equations with constant coefficients_. Homogeneous systems _Ax_ = 0 thus play a fundamental role in engineering and science.

To discuss homogeneous systems and general inhomogeneous systems we need the powerful concepts of _linear vector space_ and of _rank_. Although abstract, these concepts are not difficult.

### **2.2 LINEAR VECTOR SPACES**

We want to define a _linear vector space_ L _over a field_ F _of scalars_. First we shall give four useful instances of _fields_ and then the general definition of a _field_.

The _real numbers_ constitute a field. These are all the positive and negative numbers and zero, e.g., 0, ∑1.3,  , − _π_.

The _rational numbers_ constitute a field. These are zero and the quotients of nonzero integers, e.g., 0, −4/5, 194/13, 65, −224/95.

The _complex numbers ρ_ \+ _iσ_ ( _ρ_ and _σ_ real) constitute a field.

The two _numbers_ 0,1 _operating modulo_ 2 constitute a field. Here we define

This field is used in the theory of electronic communication.*

A _field_ , _F_ , is defined to be a set of elements for which there are operations + and • defined with these properties:

For every _α_ and _β_ in _F_ , there is an element _α_ \+ _β_ in _F_. The operation + must satisfy these laws:

(i) _α_ \+ _β_ = _β_ \+ _α_ (commutativity)

(ii) ( _α_ \+ _β_ ) + _γ_ = _α_ \+ ( _β_ \+ _γ_ ) (associativity)

(iii) There exists a unique element0 in _F_ such that _α_ \+ 0 = _α_ for all _α_ in _F_.

(iv) For every _α_ there
is a unique element – _α_ such that _α_ \+ (– _α_ ) = 0

For every _α_ and _β_ in _F_ there is an element _α_ \+ _β_ in _F_. The operation + must satisfy these laws:

(i) _α_ \+ _β_ = _β_ \+ _α_ (commutativity)

(ii) ( _α_ \+ _β_ ) + _γ_ = _α_ \+ ( _β_ \+ _γ_ ) (associativity)

(iii) There exists a unique element 1 in _F_ such that _α_ \+ 1 = _α_ for all _α_ in _F_.

(iv) For every _α except_ 0 there is a unique inverse _α_ –1 such that _α_ \+ _α_ –1 = 1.

The two operations are connected by the distributive law:

(v) ( _α_ \+ _β_ ) + _γ_ = _α_ \+ _γ_ \+ _β_ \+ _γ_

These laws are familiar to the reader for the real numbers, for the rational numbers, and for the complex numbers. But the student unfamiliar with abstract algebra should now painstakingly verify each of these laws for the field consisting of 0, 1 operating modulo 2.

We can now define a _linear vector space L_ over a field _F_. We shall use the terms linear vector space, linear space, and vector space with the same meaning. The elements of the field _F_ are called _scalars_ with respect to _L_. For every two elements, _x_ and _y_ , called "vectors" in _L_ , there is a sum _x_ \+ _y_ in _L_. (Vector addition _x_ \+ _y_ must not be confused with scalar addition _α_ \+ _β_ for elements _α_ , _β_ of the field _F_.) Vector addition is required to satisfy these laws:

(i) _x_ \+ _y_ = _y_ \+ _x_ (commutativity)

(ii) ( _x_ \+ _y_ ) + _z_ = _x_ \+ ( _y_ \+ _z_ ) (associativity)

(iii) There exists a unique vector **0** in _L_ (not to be confused with the scalar 0) such that _x_ \+ **0** = _x_ for all _x_ in _L_.

(iv) For every vector _x_ there is a unique vector – _x_ such that _x_ \+ (– _x_ ) = **0**

For every _α_ in _F_ and every _x_ in _L_ there is supposed to be a vector _αx_ in _L_. We require

(v) _α_ ( _βx_ ) = ( _αβ_ ) _x_ (associativity)

for all scalars _α_ , _β_ and vectors _x_

(vi) l _x_ = _x_ ,

where 1 is the unit scalar, for all vectors _x_

(vii) _α_ ( _x_ \+ _y_ ) = _αx_ \+ _αy_ (distributivity)

for _α_ in _F_ ; _x_ and _y_ in _L_

(viii) ( _α_ \+ _β_ ) _x_ = _αx_ \+ _βx_ (distributivity)

for _α_ , _β_ in _F_ ; _x_ in _L_

**E XAMPLE 1.** The set of row vectors

with _real_ components _α_ 1,..., _α_ 5 is a linear vector space over the field of _real_ numbers. Here we define

**E XAMPLE 2.** The set of row vectors

with _complex_ components _α_ l,..., _α_ 5 is a linear vector space over the field of _complex_ numbers.

**E XAMPLE 3.** Consider the set _Q_ of polynomials

in a _variable ω_ , where the coefficients _α_ 0, _α_ 1, _α_ 2 lie in any field _F_ , e.g., the field of real numbers. We call the zero element **0** of _Q_ the polynomial with _α_ 0 = _α_ 1 = _α_ 2 = 0. Thus, _ω_ 2 – 4 ≠ **0** (the zero polynomial) even though we do have _ω_ 2 – 4 = 0 (zero element of the field) when the variable _ω_ is replaced by ±2. The set _Q_ is a linear vector space if we define

and define

**E XAMPLE 4.** Let _F_ be the field consisting of 0 and 1 operating modulo 2. Let _L_ be the set of formal power series

where _α_ 0, _α_ 1,...lie in _F_. The set _L_ is a linear vector space under the obvious operations. For example,

**E XAMPLE 5.** Let _F_ be the field of complex numbers. Let _α_ 0, _α_ 1, _α_ 2 be in _F_. Consider the set of polynomials

of _exact_ degree 2. These polynomials do _not_ constitute a linear vector space. Why not?

**E XAMPLE 6.** Let _α_ 0, _α_ 1,..., _β_ 1, _β_ 2, . . . be real. Let ∑ | _α_ _v_ | < ∞ and ∑ | _β_ _v_ | < ∞. The set of functions of _t_ :

constitutes a linear vector space. What is the element _x_ ( _t_ )= 0 in this vector space?

**E XAMPLE 7.** Let _N_ be fixed. Let _γ_ _n_ be complex numbers ( _n_ = 0, ±1,...,± _N_ ). The set of functions

is a linear vector space over the field of complex numbers.

**E XAMPLE 8.** Consider all real-valued solutions _x_ = _ϕ_ ( _t_ ) of the linear, homogeneous differential equation

The zero element _x_ = **0** is the trivial solution _ϕ_ ( _t_ ) ≡ 0. The set of elements _x_ is a linear vector space over the field of real numbers. Thus, if _ϕ_ 1( _t_ ) and _ϕ_ 2( _t_ ) are solutions, another solution is – 7.4 _ϕ_ 1( _t_ ) + _πϕ_ 1( _t_ ).

**E XAMPLE 9.** Consider the set _L_ of column-vectors

where _x_ 1 and _x_ 2 are real numbers satisfying the constraint 3 _x_ 1 \+ 4 _x_ 2 = 0. The set _L_ is a linear vector space over the field of real numbers. Note that if _x_ and _y_ both satisfy the constraint, so does the sum _x_ \+ _y_.

Observe the graph of _L_ in Figure 2.1. In a sense which we shall later make precise, the set _L_ is one-dimensional. But every member _x_ in _L_ is represented by _two_ components, _x_ 1 and _x_ 2. Thus, _the number of parameters used to represent a vector space does_ NOT _necessarily equal the dimension of the space_.

Figure 2.1

**E XAMPLE 10.** Let _A_ be an _m_ × _n_ matrix of complex numbers. Let _x_ range over all vectors with _n_ complex components _x_ 1,..., _x_ _n_. Let _L_ be the set of all vectors _y_ of the form _y_ = _Ax_. Then _L_ is a linear vector space over the field of complex numbers. For example, if

then _L_ is the set of vectors _y_ of the form

for some complex _x_ 1, _x_ 2, and _x_ 3. Can you show that _L_ is just the space of vectors _y_ = col ( _y_ 1, _y_ 2) satisfying the constraint _y_ 2 = _iy_ 1?

**E XAMPLE 11.** Let _A_ be an _m_ × _n_ matrix of real numbers. Let _L_ be the set of vectors _x_ with _n_ real components _x_ 1, . . . , _x_ n for which _Ax_ = 0. The set _L_ is a linear vector space over the real numbers _α_. Note that _Ax_ = 0 and _Ay_ = 0 imply that _A_ ( _αx_ \+ _βy_ ) = 0.

**E XAMPLE 12.** Let _A_ be an _m_ × _n_ matrix of real numbers, and let _b_ be a column-vector with _m_ real components. Consider the set _S_ of _all_ solutions _x_ of the equation _Ax_ = _b_. Unless _b_ = 0, the set _S_ is _not_ a linear vector space because _S_ does not contain the vector _x_ = **0**. But suppose that _Ax_ = _b_ has at least one solution, say _x_ = _x_ 0. Then for all other solutions _x,_ we have

Therefore, x − x0 _lies in the linear vector space_ L _of all solutions_ y _of the homogeneous equation_ Ay = 0. For instance, the solutions _x_ of

are the vectors

where

**E XAMPLE 13.** Let _L_ be the vector space of column vectors _x_ with real components _x_ 1, . . . , _x_ 6. Let _L_ 1 consist of the vectors in _L_ for which _x_ 3 − 9 _x_ 5 = 0. Let _L_ 2 consist of the vectors in _L_ 1 for which _x_ 1 \+ _x_ 2 \+ _x_ 6 = 0. The space _L_ 1 is a _subspace_ of _L_. The space _L_ 2 is a subspace of _L_ 1, and _L_ 2 is also a subspace of _L_. We also speak of _L_ as a subspace of itself.

In this book we shall study only linear spaces, _L_ , that are subspaces of vectors written in the form

The only fields that we shall use are the field _R_ of real numbers and the field _C_ of complex numbers. If the components _x_ 1, . . . , _x_ _n_ range over all complex numbers, the vectors (15) comprise the vector space _E_ _n_ ( _C_ ) over the field _C_. If the components _x_ 1, . . . , _x_ _n_ range over all real numbers, the vectors (15) comprise the vector space _E_ _n_ ( _R_ ) over the field _R_.

When no misunderstanding can occur, we shall simply refer to _E_ _n_ as _the Euclidean space of_ n- _component column-vectors_ x. Properly, _E_ _n_ should designate _either_ En(R) _or_ En(C). If the components _x_ 1, . . . , _x_ _n_ are restricted to being real, then the scalars _α_ must also be real, and _E_ _n_ designates _E_ _n_ ( _R_ ). If the components _x_ 1, . . . , _x_ _n_ may be complex, then the scalars _α_ are also allowed to be complex, and _E_ _n_ denotes _E_ _n_ ( _C_ ).

The problems illustrate that all "finite-dimensional" vector spaces can be represented, with no loss of information, by coordinate-vectors (15). (The term _finite dimensional_ will be defined in Section 2.3). Problem 12 illustrates that linear operations on finite-dimensional vector spaces can always be represented by _matrices_. The study of linear vector spaces defined over fields different from _R_ and _C_ belongs to abstract algebra. The study of linear vector spaces of infinite dimension, illustrated in Examples 4 and , belongs to _functional analysis_.

#### **PROBLEMS**

**1.** * Show that there is no number _α_ in the field of rational numbers, for which _α_ · _α_ = 2. [Hint: Let _α_ = _p_ / _q_ , where _p_ and _q_ are integers with no common divisor. From the equation _p_ 2 = 2 _q_ 2 deduce that _p_ and _q_ must both be even. Contradiction!)

**2.** Let _V_ be the set of quantities ( _x_ 1, _x_ 2), where _x_ 1 and _x_ 2 are any _complex_ numbers. Let _F_ be the field _R_ of _real_ numbers. Can _V_ be defined as a linear vector space over _R_?

**3.** Let _W_ be the set of quantities ( _x_ 1, _x_ 2), where _x_ 1 and _x_ 2 are any _real_ numbers. Is _W_ a linear vector space over the field _C_ of complex numbers?

**4.** Show that the linear space of vectors _y_ satisfying (14) constitutes the line

**5.** Let _S_ consist of all solutions of the inhomogeneous differential equation _ϕ_ ″( _t_ ) + _ϕ_ ( _t_ ) = 1 + _t_ 2. Is _S_ a linear vector space? One solution of the equation is _x_ 0 = _t_ 2 − 1. If _ϕ_ ( _t_ ) is any solution, write _ϕ_ ( _t_ ) = _t_ 2 − 1 + _ψ_ ( _t_ ). What can you say about the functions _ψ_ ( _t_ )? (This is an analogy of Example 12.)

**6.** Show that the set of _m_ × _n_ matrices _A_ , with components in any field _F_ , is a linear _vector_ space over the field _F_. If _F_ ′ is a _sub_ field of _F_ , are the matrices _A_ a vector space over _F_ ′?

Let _F_ be a field. Let _X_ and _Y_ be different linear vector spaces defined over _F_. We speak of the spaces _X_ and _Y_ as _isomorphic_ (having the same form) if there is a one-to-one correspondence _x_ ∼ _y_ such that

**7.** Consider the linear space _X_ of all real solutions _α_ cos _t_ \+ _β_ sin _t_ of the differential equation _ϕ_ ″ + _ϕ_ = 0. Show that _X_ is _isomorphic_ to the space _Y_ of real vectors ( _α_ , _β_ ).

**8.** Show that the space _X_ of functions

is isomorphic to the space _Y_ of vectors

**9.** Show that the correspondence

is _not_ an isomorphism. (Is this correspondence one-to-one?)

**10.** Show that the correspondence

where _α_ 0, _α_ 1, _α_ 2 any real numbers, establishes an isomorphism.

**11.** * Generalize Problems 7–10 for abstract linear vector spaces _X_ consisting of all linear combinations

of certain basic vectors _b_ (1), . . . , _b_ (n). Assume _x_ 1, . . . , _x_ _n_ range over some field _F_.

**12.** * Let _D_ be the differential operator operating on polynomials _ϕ_ ( _ω_ ) = _α_ 0 \+ _α_ 1 _ω_ \+ · · · + _α_ 5 _ω_ 5 of degree  5. Thus, _D _ ( _ω_ ) = _α_ 1 \+ 2 _α_ 2 _ω_ \+ 3 _α_ 3 _ω_ 2 \+ 4 _α_ 4 _ω_ 3 \+ 5 _α_ 5 _ω_ 4. Show that _D_ is a _linear operator_ , satisfying

Establish the isomorphism

of the space _X_ of functions _ϕ_ and the space _Y_ of column vectors _y_ = col ( _α_ 0, . . . , _α_ 5). Show that the linear operator _D_ on _X_ can be _represented_ by a 6 × 6 matrix _A_ operating on _Y_. In other words, find a 6 × 6 matrix _A_ such that, if _ϕ_ ( _ω_ ) ∼ _y_ , then _Dϕ_ ( _ω_ ) ∼ _Ay_.

### **2.3 BASIS AND DIMENSION**

Let _x_ l, . . . , _x_ _m_ be vectors in a linear space _L_. These vectors are said to be _linearly independent_ if

only when _all_ of the scalars _α_ 1, . . . , _α_ _m_ are zero. The vectors _x_ 1, . . . , _x_ _m_ are called _linearly dependent_ when the equation (1) has some solution other than _α_ 1 = · · · = _α_ _m_ = 0. A sum ∑ _α_ _j_ _x_ _j_ is called a _linear combination_ of

the vectors _x j_.

**E XAMPLE 1.** The vectors

are independent.

_Proof_. The equation (1) states

The only solution of this system is _α_ 1 = _α_ 2 = 0. Why?

**E XAMPLE 2.** The vectors

are dependent because

**E XAMPLE 3.** The vectors

are dependent because 2 _x_ 1 − _x_ 3 = 0. Note here that _α_ 2 = 0. Dependence does not require that all _all α_ _j_ be nonzero in (1).

**Lemma 1.** _The vectors_ x1, . . . , xm _are dependent if and only if one of these vectors is some linear combination of the others_.

_Proof_. If _x_ 1, . . . , _x_ _m_ are dependent, then (1) holds with some scalars _α_ 1, . . . , _α_ _m_ that are not all zero. If _α_ _i_ ≠ 0, then

where _β_ _j_ = − _α_ _j_ / _α_ _i_ ( _j_ ≠ _i_ ). Conversely, an expression (2) of _x_ _i_ as a linear combination of the other vectors _x j_ yields an equation of dependence (1) with

Note in Example 3 that _x_ 1 is a combination of _x_ 2 and _x_ 3 ; _x_ 3 is a combination of _x_ 1 and _x_ 2 ; but _x_ 2 is _not_ a combination of _x_ l and _x_ 3.

The vectors _y_ l, . . . , _y_ _k_ in _L_ are said to _span_ a linear subspace _M_ if every vector _x_ in _M_ is some linear combination of _y_ l, . . . , _y_ _k_.

**E XAMPLE 4.** The independent vectors

span a certain plane through the origin in three-space. The _same_ plane is spanned by the dependent vectors

In fact, all linear combinations

are linear combinations _β_ 1 _y_ 1 \+ _β_ 2 _y_ 2 or _y_ l and _y_ 2 because

Thus _α_ 1 _y_ 1 \+ · · · + _α_ 4 _y_ 4 = _β_ 1 _y_ 1 \+ _β_ 2 _y_ 2 if we set

Vectors _b_ l, . . . , _b_ _m_ are said to be a _basis_ for a linear space _L_ if the vectors _b j_ lie in _L_ , if they are independent, and if they span _L_.

**E XAMPLE 5.** In Example 4, the vectors _y_ l, _y_ 2 are a basis for the plane that they span. The vectors _y_ l, _y_ 2, _y_ 3, _y_ 4 are not a basis because they are dependent. Another basis for this plane is

These vectors are a basis for the plane spanned by _y_ l and _y_ 2 because _z_ 1 and _z_ 2 are independent and

Thus, all vectors _α_ 1 _y_ 1 \+ _α_ 2 _y_ 2can be expressed in the form _β_ 1 _z_ 1 \+ _β_ 2 _z_ 2.

In the last example the basis _y_ l, _y_ 2 and the basis _z_ 1, _z_ 2 both consisted of two vectors. It is intuitively clear that _every_ basis must contain exactly two vectors. If this is so, we are justified in saying that _the dimension of the space is the number of vectors in_ any _basis for the space_.

**Theorem 1.** _Let_ a1, . . . , ar _be a basis for a linear vector space_ , L. _Let_ b1, . . . , bs _be another basis. Then_ r = s.

_Proof_. The proof rests on the following assertion: _If_ p _independent vectors_ xl _are linear combinations_

_of_ q _vectors_ yj, _then_  . This is obvious if _p_ = 1 because _x_ 1 ≠ 0. If _p_ > 1, we proceed by induction. The vector _x_ _p_ is given by (4) when _i_ = _p_. At least one of the coefficients _ξ_ _p_ l, . . . , _ξ_ _pq_ is nonzero because _x_ _p_ ≠ 0. Without loss of generality, suppose _ξ_ _pq_ ≠ 0. Then we have

If (5) is used in (4) when _i_ = 1, . . . , _p_ − 1, we find

where

But the _p_ − 1 vectors, _x_ _i_ − _α_ _i_ _x_ _p_ are independent, since an equation

implies

which implies that _x_ 1, . . . , _x_ _p_ are dependent. By induction from (6), we conclude that  , and hence  .

In our theorem, the _r_ independent vectors _a_ _i_ are linear combinations of the _s_ vectors _b_ _j_ ; therefore,  . Conversely, the _s_ independent vectors _b_ _i_ are linear combinations of the _r_ vectors _a_ _j_ ; therefore,  . In summary, _r_ = _s_.

**E XAMPLE 6.** The space of polynomials of degree  _n_ has dimension _n_ \+ 1 because 1, _ζ_ , . . . , _ζ_ _n_ is a basis. By Theorem 1, every other basis also must consist of _n_ \+ 1 polynomials.

**E XAMPLE 7.** The linear space of polynomials _every_ degree has no finite basis. Therefore, its dimension is not defined.

**Theorem 2.** _If_ n _vectors lie in a linear space_ L _of dimension_ m < n, _then the_ n _vectors are dependent_.

For example, three vectors in the two-dimensional plane are necessarily dependent.

_Proof_. If _n independent_ vectors are linear combinations of basis vectors _b_ 1, . . . , _b_ _m_ then  , as we showed in the proof of Theorem 1.

**Theorem 3.** _Every set of_ n _independent vectors in an_ n- _dimensional linear space, is a basis for the space_.

_Proof_. Let _a_ 1, . . . , _a_ _n_ be independent. Let _b_ 1, . . . , _b_ _n_ be a basis. By the last theorem, each set _a_ 1, . . . , _a_ _n_ , _b_ _i_ is dependent. Then there is a linear combination

with not all coefficients equal to zero. If _β_ _i_ = 0, then _α_ _i_ 1 = · · · = _α_ _in_ = 0 by the independence of the _a_ 's. Therefore, _β_ ≠ 0, and each _b_ _i_ can be found as a linear combination of the _a_ 's. The _a_ 's, therefore, span the whole space. Since they are independent, they are a basis for the whole space.

**E XAMPLE 8.** For three-dimensional Euclidean space, any three vectors in the space which are not in the same plane are a basis.

#### **PROBLEMS**

**1.** Let _A_ be an _m_ × _n_ matrix. Show that _Ax_ = 0 has a solution _x_ ≠ 0 if and only if the columns of _A_ are dependent.

**2.** Let _L_ be the set of vectors _x_ in _E_ 4 for which _x_ 1 \+ _x_ 2 \+ _x_ 3 \+ _x_ 4a = 0. Find a basis for _L_. What is the dimension of _L_?

**3.** Let _x_ and _y_ lie in the space _E_ _n_ ( _R_ ) of column-vectors with _n_ real components. These vectors are said to be _orthogonal_ if _x_ 1 _y_ 1 \+ · · · + _x_ _n_ _y_ _n_ = 0. Let _x_ l, . . . , _x_ _k_ be nonzero vectors in _E_ _n_ ( _R_ ). Let every two of these vectors be orthogonal. Show that the vectors _x_ l, . . . , _x_ _k_ are linearly independent.

**4.** Let _L_ be the linear space of solutions of the differential equation _ϕ_ ″( _t_ ) + _ϕ_ ( _t_ ) = 0. Find a basis for this space. What is the dimension of the space?

**5.** Let _L_ be the linear space of finite trigonometic sums

where _x_ 1, . . . , _x_ _k_ are real coefficients. Show that the "vectors" _x_ 1 = sin _t_ , . . . , _x_ _k_ = sin _kt_ constitute a basis for _L_. To prove independence, use Euler's identity

**6.** Find a basis for _E_ _n_. What is the dimension of _E_ _n_?

**7.** Consider the vector space of _m_ × _n_ matrices with complex components over the field of complex numbers. What is a basis for this space? What is the dimension?

**8.** Consider the vector space of _m_ × _n_ matrices with complex components over the field of _real_ numbers. Show that the dimension of this space is 2 _mn_.

**9.** Let _a_ 1, . . . , _a_ _n_ be dependent vectors in _L_. Assume that at least one _a_ _j_ ≠ 0. Show that there is a linearly independent subset _a_ _j_ 1, _a_ _j_ 2, . . . , _a_ _j_ _r_ of which every other _a_ _k_ is some linear combination. (If _a_ 1, . . . , _a_ _n_ are the columns of a matrix _A_ , the integer _r_ is called the _rank_ of _A_ ).

**10.** Let the positive integer _N_ be fixed. Let _L_ be the space of real-valued functions _ϕ_ ( _t_ ) defined at the times _t_ = 0, ±1/ _N_ , ±2/ _N_ , . . . and satisfying the condition of periodicity _ϕ_ ( _t_ \+ 1) = _ϕ_ ( _t_ ). What is the dimension of _L_? Give a basis for _L_.

### **2.4 SOLVABILITY OF HOMOGENEOUS EQUATIONS**

We are now ready to prove the result which, as we shall see later, is the basis for the theory of eigenvalues and eigenvectors.

**Theorem 1.** _Let_ A _be an_ n × n _matrix. Then the equation_ Ax = 0 _has a solution_ x ≠ 0 _if and only if det_ A = 0.

_Proof_. We know already, from Section 1.6, that det _A_ ≠ 0 implies that _A_ has an inverse _A_ −1. Then _Ax_ = 0, multiplied on the left by _A_ −1, yields _x_ = 0.

Therefore, we have only to prove that det _A_ = 0 implies the existence of a nonzero solution _x_ to the equation _Ax_ = 0. Let _a_ 1, . . . , _a_ _n_ be the columns of _A_. Let _x_ 1, . . . , _x_ _n_ be the components of _x_. Then

_Thus_ , Ax = 0 _has a solution_ x ≠ 0 _if and only if the columns of_ A _are dependent_.

The columns _a_ 1, . . . , _a_ _n_ lie in the space _E_ _n_ of column vectors with _n_ components. The dimension of this linear space is _n_ because it has a basis consisting of the _n_ vectors

The vectors (2) clearly span _E_ _n_. They are independent because a linear combination of the _e_ _i_ with coefficients _c_ _i_ is simply the column-vector with components ct, which is zero only if all _c_ _i_ = 0.

By Theorem 3 of the last section, if the _n_ vectors _a_ 1, . . . , _a_ _n_ are independent, they are a basis for the _n_ -dimensional space _E_ _n_. Then every vector _b_ in _E_ _n_ is a linear combination

In other words, the equation _Ay_ = _b_ is solvable for every _b_. Then det _A_ ≠ 0, by Theorem 2 of Section 1.7. Therefore, det _A_ = 0 implies that the columns _a_ _j_ are dependent, i.e., that _Ax_ = 0 for some _x_ ≠ 0.

**Theorem 2.** _If A is an_ m × n _matrix with_ m < n, _then_ Ax = 0 _has a solution_ x ≠ 0.

**E XAMPLE 1.** The two equations in three unknowns

must have a solution with not all _x_ _i_ = 0.

_Proof_. The columns of _A_ are _n_ vectors in the space _E_ _m_ of dimension _m_ < _n_. Therefore, the columns are dependent, by Theorem 2 of Section 2.3. Thus,

for some _x_ ≠ 0.

If _Ax_ = 0 for some _x_ ≠ 0, it is natural to ask, in some sense, how many solutions there are. If _x_ is a solution ≠ 0, so is λ _x_ for all λ ≠ 0. Can _Ax_ = 0 have linearly independent solutions? The set of solutions _x_ to _Ax_ = 0 is known as the _null space_ of _A_.

**E XAMPLE 2.** The null space of

consists of all vectors _x_ of the form

**Theorem 3.** _The null space of an_ m × n _matrix is a linear vector space of dimension_  n.

_Proof_. If _Ax_ = 0 and _Ay_ = 0, then _A_ ( _αx_ \+ _βy_ ) = 0. Therefore, the null space is a linear vector space. If the null space contains _k_ independent vectors, then  , because these vectors are also independent vectors in the including space _E_ _n_ , which has dimension _n_. If _x_ = 0 is the only vector in the null space, its dimension is zero. Otherwise, let   be the largest integer such that the null space contains _k_ independent vectors, say _b_ 1, . . . , _b_ _k_. Then these vectors are a basis for the null space because, for any _x_ in the space, we have the dependence

Now _α_ 0 ≠ 0 for _α_ 0 = 0 implies that the _b_ 's are dependent. Hence, _x_ is a linear combination of the independent vectors _b_ _j_. Therefore, the null space has dimension  .

In the context of this proof we have proved the obvious but useful result: _If a linear space_ L _is included in a linear space_ L1 _of dimension_ n, _then_ L _has a dimension_  n.

In Example 2 we note that the null space has dimension 2. In fact,

provide a basis. The matrix _A_ has dependent columns; all three columns of _A_ are multiples of the first column. _We define the rank of a matrix as the dimension of the linear space spanned by its columns_.

**E XAMPLE 3.** We have

because the linear space of all combinations

has col (1, 2) as a basis.

**Theorem 4.** _The rank of_ A _is the largest integer_   _such that_ A _has_ r _independent columns_.

_Proof_. Let _L_ be the linear space spanned by the columns of _A_. Suppose

that _A_ has _r_ independent columns, for example, _a_ 1, . . . , _a_ _r_. If _r_ = _n_ , the dimension of _L_ is _n_.

Suppose  . If every _r_ \+ 1 columns of _A_ are dependent, the _r_ \+ 1 vectors _a_ 1, . . . , _a_ _r_ , _a_ _s_ are dependent for any _s_. Let

Then _α_ 0 ≠ 0 because _a_ 1, . . . , _a_ _r_ are independent. Therefore, (8) may be solved for _a_ _s_ as a linear combination of _a_ 1, . . . , _a_ _r_. Since every column of _A_ is a linear combination of _a_ 1, . . . , _a_ _r_ , all the linear combinations of the columns of _A_ are linear combinations of just the _r_ columns _a_ 1, . . . , _a_ _r_. Therefore, these _r_ independent columns are a basis for _L_. Therefore, the dimension of _L_ is _r_.

If _r_ = 0 is the largest number of independent columns, then _A_ is the zero matrix, and the dimension of _L_ equals zero. This completes the proof. We can now evaluate the dimension of the null space of _A_ in terms of the rank of _A_.

**Theorem 5**. _The vectors_ x _solving_ Ax = 0 _comprise a linear space with a dimension which is the number of columns of_ A _minus the rank of_ A.

**E XAMPLE 4.** Let _A_ be defined as in the last two examples. The null space has dimension 2, the rank is 1, and the number of columns is 3.

_Proof of the Theorem_. Let _A_ be an _m_ × _n_ matrix with rank _r_. If _r_ = _n_ , the columns of _A_ are independent; _x_ = 0 is the null space, with dimension 0 = _n_ − _r_. If _r_ = 0, then _A_ = 0 the null space is all _E_ _n_ , with dimension _n_ = _n_ − _r_.

Suppose  . Without loss of generality, suppose that the first _r_ columns of _A_ , namely _a_ 1, . . . , _a_ _r_ , are independent. Let _x_ 1, . . . , _x_ _v_ be a basis for the null space. We want to show that _v_ = _n_ − _r_.

If _e_ 1, _e_ 2, . . . are the unit vectors in _E_ _n_ , then _a_ 1 = _Ae_ 1, _a_ 2 = _Ae_ 2, . . . . We assert that the _r_ \+ _v_ vectors _e_ 1, . . . , _e_ _r_ , _x_ 1, . . . , _x_ _v_ are independent. Suppose

Since each _x_ _j_ lies in the null space, we have _Ax_ _j_ = 0. Therefore, multiplication of (9) by _A_ yields

Now the independence of _a_ 1, . . . , _a_ _r_ implies all _α_ _i_ = 0. Equation (9) now becomes Σ _β_ _j_ _x_ _j_ = 0, which implies all _β_ _j_ = 0, by the independence of the basis vectors _x_ _j_.

Having shown that the _r_ \+ _v_ vectors _e_ 1, . . . , _e_ _r_ , _x_ 1, . . . , _x_ _v_ are independent, we now show that they span _E_ _n_ , hence that they are a basis for _E_ _n_ , hence that _r_ \+ _v_ = _n_. For any column _a_ _s_ there is a representation

because _a_ 1, . . . , _a_ _r_ are a basis for the space spanned by the columns of _A_. Since _a_ _i_ = _Ae_ _i_ , (11) states that

Thus, _x_ is in the null space. Therefore, _x_ is a linear combination of the basis vectors _x_ 1,. . ., _x_ _v_. Therefore, _e_ _s_ is a combination of _e_ 1, . . . , _e_ _r_ and _x_ 1, . . . , _x_ _v_. This is true for all the _n_ unit vectors _e_ _s_ that span _E_ _n_. Therefore, _e_ 1, . . . , _e_ _r_ , _x_ 1, . . . , _x_ _v_ span _E_ _n_ the proof is done.

**E XAMPLE 5.** Let _A_ be defined as in the preceding examples. A basis for the null space is given by (6). Now

form a basis for _E_ 3.

#### **PROBLEMS**

**1.** Let _A_ be an _n_ × _n_ matrix. In terms of a determinant, give the necessary and sufficient condition on the scalar λ such that the equation _Ax_ = λ _x_ has a solution _x_ ≠ 0.

**2.** Let

What is the null space of _A_? What is the rank of _A_? What is the dimension of the null space? Which sets of columns of _A_ are bases for the space spanned by all of the columns?

**3.** Answer the questions of Problem 2 for the matrix

**4.** Prove that rank ( _A_ \+ _B_ )   rank ( _A_ ) + rank ( _B_ )

**5.** Prove that rank ( _AB_ )   rank _B_. Prove also that rank ( _AB_ )   rank _A_ , and hence, rank ( _AB_ )   min (rank _A_ , rank _B_ ).

**6.** Let _Ā_ be the matrix ( _ā_ _μv_ ) formed by taking the complex conjugates of the components of _A_. Show that rank _Ā_ = rank _A_.

**7.** Define _A_ * = ( _Ā_ ) _T_ = the transpose of _Ā_. In particular, if _x_ is a column-vector, define _x_ * to be the row-vector formed from the conjugates of the components of _x_. For any vector _y_ , define || _y_ ||2 = Σ || _y_ _μ_ ||2 = _y_ * _y_. Show that _x_ * _A_ * = ( _Ax_ )*. Hence, show that _A_ * _Ax_ = 0 implies || _Ax_ ||2 = 0, which implies _Ax_ = 0. Deduce that _A_ * _A_ and _A_ have the _same_ null space.

**8.** From the result of the last problem, and from Theorem 5, deduce that _A_ * _A_ and _A_ have the same rank. From the result of Problem 5, prove that rank _A_   rank _A_ *. By replacing _A_ by _A_ * [what is ( _A_ *)*?], prove that rank _A_ *   rank _A_ , and hence rank _A_ = rank _A_ *. From the result of Problem 6, show that rank _A_ = rank ( _A_ _T_ ). Deduce that the rank of a matrix _A_ equals the dimension of the linear space spanned by its _rows_.

**9.** From the result of Problem 7, show that the vectors _a_ 1, _a_ 2, . . . , _a_ _n_ in   are independent if and only if det ( _A_ * _A_ ) ≠ 0, where _A_ is the _m_ × _n_ matrix with the columns _a_ 1, . . . , _a_ _n_. The determinant of the square matrix _A_ * _A_ is known as the _Gram determinant_.

### **2.5 EVALUATION OF RANK BY DETERMINANTS**

Given an _m_ × _n_ matrix _A_ , we wish to find the rank of _A_ by observing the determinants of square submatrices. A _submatrix_ is defined as a square matrix formed from certain columns and rows of _A_.

**E XAMPLE 1.** The matrix

is the submatrix formed from rows 3, 5, 8 and columns 2, 4, 8 of the rectangular matrix _A_ = ( _a_ _ij_ ) ( _i_ = 1, . . . , 14; _j_ = 1, . . . , 93).

We may speak of a submatrix _R_ of a submatrix _S_.

**E XAMPLE 2.** _S_ is a submatrix of itself. Other submatrices of the submatrix _S_ defined in (1) are

**E XAMPLE 3.** Let

We showed in the last section that rank _A_ = 1. Observe that the 2 × 2 submatrices

all have zero determinants, but that there is a 1 × 1 submatrix of _A_ that has a nonzero determinant.

**Theorem 1.** _Let_ A _be an_ m × n _matrix_. _Suppose that_ A _has an_ r × r _submatrix_ S _with det_ S ≠ 0. _And suppose that every_ (r + 1) × (r + 1) _submatrix_ T _of which_ S _is a submatrix_ , _has det_ T = 0. _Then rank_ A = r.

**E XAMPLE 4.** Let

The submatrix

has det _S_ ≠ 0. But every 3 × 3 submatrix _T_ including _S_ has det _T_ = 0. For example,

Theorem 1 now states that rank _A_ = 2.

_Proof of the Theorem_. For definiteness, suppose that _S_ is the _r_ × _r_ submatrix in the upper left corner of _A_ , i.e.,

It will be apparent in the proof that this specification involves no loss of generality.

Let _a_ 1, . . . , _a_ _r_ , . . . , _a_ _n_ be the columns of _A_. Let _b_ _i_ , . . . , _b_ _r_ , . . . , _b_ _n_ be the columns of the _r_ × _n_ matrix formed from the first _r_ rows of _A_. Thus, if _r_ = 2 and

We assert that the columns _a_ 1, . . . , _a_ _r_ are independent. This is true if the columns _b_ 1, . . . , _b_ _r_ are independent because

But the columns _b_ 1, . . . , _b_ _r_ form the submatrix _S_ in formula (8). Since det _S_ ≠ 0, _Sz_ = 0 implies _z_ = 0, and the columns of _S_ are independent.

It only remains to show that the independent columns _a_ 1, . . . , _a_ _r_ span the other columns of _A_ if _n_ > _r_. Let _s_ > _r_. Since det _S_ ≠ 0, _b_ _s_ is some linear combination of _b_ 1, . . . , _b_ _r_ :

In the example (9), if _s_ = 4, formula (11) becomes

We now assert that _a_ _s_ is a linear combination of _a_ 1, . . . , _a_ _r_ with the same coefficients:

In our example, we assert that

Suppose that (13) were false. Then for some lower-row component _a_ _ts_ with _t_ > _r_ we must have inequality:

We now form the ( _r_ \+ 1) × ( _r_ \+ 1) submatrix

The given matrix _S_ is the _r_ × _r_ submatrix formed by deleting the last row and column of _T_. We will show that (15) implies det _T_ ≠ 0, contrary to hypothesis.

Formulas (11) and (15) show that the computation of det _T_ may be simplified by subtracting the multiples _β_ 1 . . . , _β_ _r_ of the first _r_ columns of _T_ from the last column of _T_. The last column of the resulting matrix, _T′_ , consists of _r_ zeros followed by the nonzero difference, _δ_ , between the two sides of the inequality (15).

Expansion of det _T′_ by the last column gives

Since _T_ is an ( _r_ \+ 1) × ( _r_ \+ 1) submatrix including _S_ , we must have det _T_ = 0. The contradiction (18) shows that the inequality (15) does not occur. This completes the proof.

Theorem 1 gives a construction for the rank: Look for any nonzero component _a_ _Pq_. If there is none, rank _A_ = 0. Let _S_ 1 be the 1 × 1 matrix _a_ _pq_. Look for a 2 × 2 matrix _S_ 2 including _S_ 1 with det _S_ 2 ≠ 0. If no such _S_ 2 exists, rank _A_ = 1. Proceed until an _r_ × _r_ matrix _S_ _r_ is formed including _S_ _r_ -1 such that det _S_ _r_ ≠ 0, where _r_ is as large as possible. Then rank _A_ = _r_.

**E XAMPLE 5.** Let

Then we may take

Now no 3 × 3 _S_ 3 can be found in _A_ such that _S_ 3 includes _S_ 2 and det _S_ 3 ≠ 0. Therefore, _r_ = 2 is as large as possible. Note that another chain of submatrices, e.g.,

would also yield _r_ = 2. Note also that we need not look at _all_ 3 × 3 submatrices; we need only to look for those that include _S_ 2.

**Theorem 2.** _The rank of a matrix_ A _is the largest integer_ r _such that_ A _has an_ r × r _submatrix_ Sr _with det_ Sr ≠ 0.

_Proof_. Every ( _r_ \+ 1) × ( _r_ \+ 1) submatrix _T_ including _S_ _r_ has det _T_ = 0. The result now follows from Theorem 1.

We have defined rank as the dimension of the space spanned by the columns. It is a remarkable fact that the space spanned by the rows has the _same_ dimension.

**Theorem 3.** Rank _A_ = rank _A_ _T_.

_Proof_. Since the submatrices of _A_ _T_ are the transposes of the submatrices of _A_ , the result follows from Theorem 2.

**E XAMPLE 6.** The 3 × 7 matrix in formula (5) has just two independent rows. _Therefore_ , it has just two independent columns, and the rank equals 2.

In the problems for Section 2.4 a proof that rank _A_ = rank _A_ _T_ was outlined which does not use determinants.

#### **PROBLEMS**

**1.** Find the rank of

What can you choose for the submatrix _S_ described in Theorem 1?

**2.** When the submatrix _S_ described in Theorem 1 has been found, how can one find a set of columns of _A_ which form a basis for the space spanned by _all_ the columns of _A_?

**3.** Prove that the rank of a matrix is not changed if two rows are interchanged, if a row is multiplied by a nonzero scalar, or if a scalar multiple of one row is added to another row.

### **2.6 THE GENERAL m × n INHOMOGENEOUS SYSTEM**

**Theorem 1.** _Let_ A _be an_ m × n _matrix_. _Then the_ m _equations in_ n _unknowns_

_have a solution_ x _if and only if the_ m × (n + 1) _matrix_ B _formed by adjoining the extra column_ b _to the right of_ A, _has rank_ B = _rank_ A. _If this condition for solvability holds, then the set of all solutions_ x _consists of any particular solution_ x0 _plus a linear space of dimension_ n − _rank_ A.

**E XAMPLE 1.** Consider three equations in two unknowns:

In this example,

If _b_ = col (1, 0, 0), then (2) has no solution because rank _B_ = 2 > rank _A_ = 1. If _b_ = col (5, 10, 15), then rank _B_ = 1, and every vector of the form

solves (2).

_Proof of the Theorem_. Let _L_ _A_ and _L_ _B_ be the linear vector spaces spanned, respectively, by the columns of _A_ and by the columns of _B_. Since _L_ _A_ is a subset of _L_ _B_ , we have

If rank _A_ = rank _B_ = _r_ , then _r_ independent columns of _A_ are a basis for _L_ _A_ ; and they provide a basis for _L_ _B_ because they are _r_ independent vectors in the linear space _L_ _B_ of dimension _r_. In particular, the vector _b_ in _L_ _B_ is some linear combination of the _r_ independent columns of _A_ :

Setting _x_ _j_ =0 when _a_ _j_ is not in the basis, we have a solution _x_ to the equations (1).

If rank _A_ < rank _B_ , then the subset _L_ _A_ of _L_ _B_ cannot be identical to _L_ _B_ , because then they would have the same dimension. If _b_ lay in _L_ _A_ , then _L_ _B_ would equal _L_ _A_. Therefore, _b_ lies outside _L_ _A_ , which means that _Ax_ = _b_ has no solution.

If _Ax_ = _b_ is solvable, let _x_ 0 be any particular solution. Then _A_ ( _x_ − _x_ 0) = 0. By Theorem 5 of Section 2.4, the vectors _y_ satisfying _Ay_ = 0 form a linear space of dimension _n_ − _r_. Thus, the vectors _x_ satisfying _Ax_ = _b_ are those for which _x_ − _x_ 0 = _y_ , where _y_ lies in the ( _n_ − _r_ )-dimensional null space of _A_.

**E XAMPLE 2.** Since the square matrix

has det _A_ = 0, either _Ax_ = _b_ has no solution _or Ax_ = _b_ has infinitely many solutions.

For square matrices, our results are summarized in Figure 2.2.

**Figure 2.2**

#### **PROBLEMS**

**1.** For which vectors _b_ does the system

have a solution?

**2.** Let _A_ be an _m_ × _n_ matrix, and _B_ be an _m_ × _p_ matrix. Give a necessary and sufficient condition so that the system _AX_ = _B_ will have an _n_ × _p_ matrix-solution _X_. If _X_ ° is a particular solution, and if rank _A_ = _r_ , describe all other solutions.

**3.** Give all the solutions of the system

**4.** Give all the solutions of the system

**5.** Prove that the system

has no solution. Apply Theorem 1.

### **2.7 LEAST-SQUARES SOLUTION OF UNSOLVABLE SYSTEMS**

In applied science it is common to try to explain the behavior of a variable _b_ , insofar as it depends on certain variables _a_ 1, _a_ 2, . . . , _a_ _n_ , by hypothesizing a _linear_ dependence

For example, we may hypothesize that the average yearly income, _b_ , of men at age forty depends upon the number of years of education, _a_ 1, and upon I.Q. score, _a_ 2, by some relationship

An equation (2) might also be used to describe the rate, _b_ , of a certain chemical reaction as it depends upon temperature, _a_ 1 and upon the concentration, _a_ 2, of a certain catalyst. As a third example, the Dow Jones average price, _b_ , for industrial stocks may be hypothesized to depend on time, _t_ (years), by means of a linear relationship _b_ = _x_ 0 \+ _x_ 1 _a_ 1 \+ · · · + _x_ 4 _a_ 4, where

To determine the unknown coefficients of dependence _x_ 0, _x_ 1, . . . , _x_ _n_ , many observations are made. For each observation _i_ = 1, . . . , _m_ , numerical data

are recorded. Thus, _b_ _i_ is the value of _b_ in the _i_ th observation (or experiment); _a_ _ij_ is the value of _a_ _j_ in the _i_ th experiment. We then try to solve the equations

If the number of observations, _m_ , is greater than the number of unknowns, 1 + _n_ , the linear system (4) usually has no solution. Nevertheless, one would like to find the _best possible_ values _x_ 0, . . . , _x_ _n_ for the unsolvable linear system (4).

Of the many senses in which the phrase _best possible_ can be defined, the commonest is the sense of _least squares_ : _We pick the numbers_ x0, . . . , xn _to minimize the sum of the squares of the differences:_

Differentiation of _ϕ_ with respect to _x_ _j_ gives the condition:

where we define _a_ _i_ 0 = 1. The system (6), which is called the _normal equations,_ consists of _n_ \+ 1 equations in _n_ \+ 1 unknowns.

To discuss the normal equations, we introduce the vector of data _b_ _i_ , the matrix of data _a_ _tj_ , and the vector of unknown coefficients _x_ _j_.

If _y_ is the column-vector of differences

then _y_ = _b_ − _Ax_. The normal equations (6) now take the form

or, with _A_ _T_ = the transpose of _A_ ,

or

or

The system (8) can be solved if the square matrix _A_ _T_ _A_ has det ( _A_ _T_ _A_ ) ≠ 0. But det ( _A_ _T_ _A_ ) = 0 implies that there is some solution _z_ ≠ 0 to the equation _A_ _T_ _Az_ = 0. Multiplication of this equation on the left by the row vector _z_ _T_ gives

But the reader will easily verify the identity

which holds for all matrix-vector products _Az_. Then (9) states that _w_ _T_ _w_ = 0 if _w_ = _Az_. Then

Since we have tacitly assumed that all numbers in this discussion are real, equation (11) yields _w_ = _Az_ = 0. In this case there cannot be a _unique_ best possible set of coefficients _x_ 0, . . . , _x_ _n_ for the equations (4). In matrix-vector form, these equations are written as b = _Ax_. But if _x_ is any vector, and if _ϕ_ ( _x_ ) is the sum of squares (5), and if _Az_ = 0, then

for all scalars λ. _Thus_ , _if the sum of squares ϕ is minimized by a unique vector_ x, _then_ det (ATA) ≠ 0, _and the vector_ x _is the unique solution of the normal equations_ ATAx = AT **b**.

**E XAMPLE 1.** Let us find the best linear fit

where four observations give the data

Here we have

The system _Ax_ = **b** is unsolvable because b is not a linear combination of the columns of _A_. ( _Proof_ : Let _B_ be the 4 × 4 matrix [ _A_ , **b** ]. Computation yields det _B_ = 1 ≠ 0. Therefore, the columns of _B_ are independent.) The normal equations are

for which computation yields

Thus, the unique best linear fit (12), in the least-squares sense, to the data (13) is

The reader will note that this equation does not hold exactly for _any_ of the four observations (13).

_Even if_ det (ATA) = 0, _the least-squares problem is still solvable_. We have shown that det ( _A_ _T_ _A_ ) = 0 implies that the columns of _A_ are dependent. The problem is still to choose _x_ so as to minimize

If _A_ is the zero matrix, then _Ax_ = 0 for all _x_ ; and all vectors _x_ minimize _ϕ_. If _A_ ≠ 0, let _A_ 1 be a matrix formed by successively eliminating dependent columns until only a set of independent columns remains. For example, if

Since every column of _A_ is a linear combination of the columns of _A_ 1, every linear combination, _Ay_ , of the columns of _A_ is some linear combination, _A_ 1 _w_ , of the columns of _A_ 1. Therefore, if _Ay_ = _A_ 1 _w_ ,

and

But the columns of _A_ 1 are independent! Therefore, we have

and the normal equations

have a unique solution  . A vector _y_ = _x_ minimizing _ϕ_ ( _y_ ) is now found by letting _x_ _j_ = 0 if the dependent column _a_ _j_ was deleted from _A_ , and by letting _x_ _j_ = _u_ _k_ if _a_ _j_ is the _k_ th column of the abridged matrix _A_ 1.

**E XAMPLE 2.** Let _A_ be the first matrix in (17). Let us minimize

If _A_ 1 is the second matrix in (17), the normal equations (19) become

These equations have the unique solution  ,  . Therefore, _y_ = _x_ minimizes _ϕ_ ( _y_ ) if  .

The careful reader will have observed that we never proved that a solution, _x_ , of the normal equations minimizes the sum of squares, _ϕ_. The vanishing of the first derivatives, _∂ϕ∂x_ _j_ , is a necessary but not a sufficient condition for minimization. Suppose that _x_ is a solution of the normal equations _A_ _T_ _Ax_ = _A_ _T_ **b**. If _y_ is any other vector, let _y_ = _x_ \+ _z_. Then

But

Therefore,

with equality if and only if _Az_ = 0.

#### **PROBLEMS**

**1.** Find a least-squares fit _b_ = _x_ 0 \+ _x_ 1 _a_ 1 \+ _x_ 1 _a_ 1 for the data

**2.** Suppose that we desire a least-squares fit

where there is no constant term _x_ 0. Show that the normal equations are now _A_ _T_ _Aα_ = _A_ _T_ **b** , where _A_ =( _a_ _ij_ ) _i_ = 1, . . . , _m_ ; _j_ = 1, . . . , _n_. (There is no initial column of 1's.)

**3.** * Suppose that we require a least-squares fit

where all numbers may be complex. Redefine the sum of squares _ϕ_ ( _x_ ) appropriately. Develop the normal equations _A_ * _Ax_ = _A_ * **b** , where _A_ * = ( _Ā_ ) _T_. In other words, generalize the theory in this section for _complex_ data.

**4.** Suppose that we feel that some observations are more important or reliable than others. Redefine the function to be minimized as

Now what are the appropriate normal equations? Assume real data.

**5.** Find a least-squares fit _b_ = _x_ 0 \+ _x_ 1 _a_ 1 \+ _x_ 2 _a_ 2 \+ _x_ 3 _a_ 3 \+ _x_ 4 _a_ 4 for the data

Note that det( _A_ _T_ _A_ ) = 0.

* * *

* See _Digital Communication with Space Applications_ by S. W. Golomb, L. D. Baumert, M. F. Easterling, J. J. Stiffler, and A. J. Viterbi. Englewood Cliffs, N.J.: Prentice-Hall, Inc., 1964.

## **3 MATRIX ANALYSIS OF DIFFERENTIAL EQUATIONS**

### **3.1 INTRODUCTION**

Although the theory of matrices is usually taught as a branch of algebra, matrices are a powerful tool in mathematical analysis. From a purely logical standpoint, this chapter is unnecessary to the development of the theory of matrices. Nevertheless, the study of differential equations provides an important motivation for the study of eigenvalues, eigenvectors, and canonical forms.

### **3.2 SYSTEMS OF LINEAR DIFFERENTIAL EQUATIONS**

Consider the single differential equation of fourth order

This equation is equivalent to a system of four differential equations of first order. Let

Then we have

The first three equations come from (2); the last comes from (1). The system (3) can be written with matrices and vectors in the form

where

Most systems of linear ordinary differential equations are equivalent to first-order systems (4). Consider

The highest derivative of _u_ in (5) is _u_ ″; the highest derivative of _v_ is _v_ ′. Solve for _u_ ″ and _v_ ′ in terms of the lower-order derivatives:

Set

Then

Therefore, (5) is equivalent to the first-order system (4) where

The general principle is clear: Suppose that we have _m_ differential equations for _m_ unknowns _u_ 1, . . . , _u_ _m_. Let the highest-order derivative of _u_ _i_ in the system be of order  . _If the_ m × m _matrix of coefficients of these highest_ - _order derivatives has a nonzero determinant, we may solve for the highest_ - _order derivatives in terms of the lower_ - _order derivatives_. We now define

where _n_ = _α_ 1 \+... + _α_ _m_. Now the _x_ 's may be used to form a first-order system _x_ ′ = _Ax_ \+ _f_.

#### **PROBLEMS**

**1.** Write the equation

as a first-order system _x_ ′ = _A_ ( _t_ ) _x_.

**2.** Write the system

in the form _x_ ′ = _Ax_ \+ _f_ ( _t_ ).

**3.** Can the system

be put into the form _x_ ′ = _Ax_ \+ _f_ ( _t_ )? Does this system have a solution satisfying the initial conditions _u_ (0) = 1, _v_ (0) = 2?

**4.** Let _x_ ′( _t_ ) = _A_ ( _t_ ) _x_ ( _t_ ) + _f_ ( _t_ ) for  , with _x_ (0) = 0. Let _f_ ( _t_ ) range in some linear vector space of functions of _t_. Assuming the existence and uniqueness of the solution _x_ ( _t_ ), show that the constant vector _x_ (1) depends _linearly_ on the function _f_ ( _t_ ).

**5.** Let _x_ ′( _t_ ) = _A_ ( _t_ ) _x_ ( _t_ ) for  . Show that there is a constant matrix _M_ such that _x_ (1) = _Mx_ (0) for all solutions _x_ ( _t_ ). Letting _t_ vary backwards from 1 to 0, show that _M_ has an inverse. (Assume the existence and uniqueness of the solution of the initial-value problem.)

**6.** * Assume the notation of Problem 5. Let _X_ ( _t_ ) be the _matrix_ satisfying the initial-value problem _X_ ′ = _AX_ , _X_ (0) = _I_. Show that _M_ = _X_ (1). [Hint: Write _x_ ( _t_ ) in terms of _X_ ( _t_ ) and _x_ (0).]

**7.** * Assume the notation of the last two problems. Assume that _A_ ( _t_ \+ 1) = _A_ ( _t_ ) for all _t_. Show that _x_ ′ = _A_ ( _t_ ) _x_ has a nonzero solution _x_ ( _t_ ) with _x_ ( _t_ \+ 1) ≡ _x_ ( _t_ ) if and only if the matrix _X_ (1) = _M_ satisfies the condition det ( _M_ − _I_ ) = 0.

### **3.3 REDUCTION TO THE HOMOGENEOUS SYSTEM**

In this section we will show how the inhomogeneous system of linear differential equations _x_ ′ = _Ax_ \+ _f_ can be reduced, by matrix analysis, to the homogeneous system _x_ ′ = _Ax_.

Let _e_ _j_ be the _j_ th unit vector: _e_ _j_ =( _δ_ _ij_ ) ( _i_ = 1, . . ., _n_ ). Let _u_ _j_ ( _t_ ) solve the initial-value problem

Let _X_ ( _t_ ) be the _n_ × _n_ matrix whose _j_ th column is the vector _u_ _j_ ( _t_ ). Then (1), taken for _j_ = 1, . . . , _n_ , is equivalent to the initial-value problem

For simplicity, we shall suppose that all components of _A_ ( _t_ ) are continuous.

The matrix _X_ ( _t_ ) may be called the _fundamental solution_ of the homogeneous equation (2). _For any initial vector_ , x(0) = c, _the equation_ (1) _has the solution_ x(t) = X(t)c. Methods for computing _X_ ( _t_ ) if _A_ is constant, are given in the next two sections. In this section we shall assume that _X_ ( _t_ ) is known.

To solve the inhomogeneous equation

introduce a new unknown, _z_ ( _t_ ), by the equation

Now we assert

This follows from differentiation of the identity _y_ _i_ = ∑ _X_ _ij_ _z_ _j_.

Setting (6) in the left-hand side of (4), and (5) in the right-hand side of (4), we find

Since _dX_ / _dt_ = _AX_ , we may simplify (7) to obtain

By (3), _X_ ( _t_ ) has an inverse for _t_ = 0. Assume that _X_ −1( _t_ ) exists and is continuous; the proof of this assumption is outlined in the Problems for this section. Assume that _f_ ( _t_ ) is continuous. Then

From (5) and (10) we obtain the solution

Note that _z_ (0) = _y_ (0) by (5).

**E XAMPLE 1.** We will solve the problem

We are given that (2) and (3) are satisfied by

Then

By the trigonometrical addition-formulas,

Now by (11)

Observe in (13) the curious identity

We can show that this is always true if the matrix _A_ is independent of _t_ , as it is in (12). Suppose that

Let _Y_ ( _t_ ) = _X_ ( _t_ − _τ_ ) for fixed _τ_. Evidently,

But _both_ sides of (15) satisfy (17). The identity (15) now follows from the uniqueness of the solution of the initial-value problem (17).

_We have just shown that_ , _if_ A _is independent of_ t, _the inhomogeneous equation_ y′ = Ay + f(t) _has the solution_

_provided_ X′(t) = AX(t), X(0) = I.

That (18) provides the required solution may be verified independently by direct differentiation of the integral:

The equation _y_ ′ = _Ay_ \+ _f_ follows from adding (19) to

Similarly, (11) may be verified directly.

#### **PROBLEMS**

**1.** We note that ( _t_ \+ 1)3 and ( _t_ \+ 1)−2 solve

Find the fundamental matrix solution _X_ ( _t_ ) for the system of equations

Compute the Wronskian _ω_ ≡ det _X_ and compute the inverse _X_ −1.

**2.** Solve the initial-value problem

**3.** Define the _Wronskian_ determinant

where _u_ _j_ = _u_ ( _j_ ) solves (1). Applying the rule for differentiation in Section 1.8, show that

where _ω_ _i_ is the determinant written by replacing the _i_ th row of the Wronskian _ω_ by the row

**4.** Write the result of Problem 3 explicitly for the general 2 × 2 system. Prove that the determinants _ω_ 1 and _ω_ 2 can be simplified to the forms:

**5.** For the general _n_ × _n_ system show that

**6.** Show that the Wronskian, itself, satisfies a differential equation. Deduce that the Wronskian equals

or simply _ω_ (0) · exp [(∑ _a_ _kk_ ) _t_ ] if the _a_ 's are constants. Can the Wronskian ever equal zero?

**7.** If the fundamental matrix _X_ ( _t_ ), solving (2), exists for,  , prove that the inverse matrix _X_ −1( _t_ ) exists and is continuously differentiable for  .

### **3.4 SOLUTION BY THE EXPONENTIAL MATRIX**

We have reduced systems of linear differential equations to the problem of obtaining the fundamental solution _X_ ( _t_ ) solving

From now on we shall suppose that _A_ is an _n_ × _n constant_ matrix.

If _n_ = 1, we have the familiar solution

But the right-hand side of (2) may have a meaning even if _n_ > 1. Let _A_ 0 = _I_ and, if  ,

Then the partial sums

are well-defined _n_ × _n_ matrices:

For _n_ > 1 the right-hand side of (2) makes sense if each component partial sum   tends to a limit as _N_ → ∞. We then regard (2) as a definition. For _t_ = 0, exp ( _At_ ) has the value _I_. Differentiation of (2) term by term yields

This procedure is valid if the resulting infinite series of matrices converges uniformly for _t_ in any bounded interval.

**Theorem 1.** _Let A be an_ n × n _matrix_. _Then the infinite series of matrices_

_converges uniformly and absolutely for_ t _in any bounded interval_ , _and provides the solution_ X(t) _to the initial-value problem_ X′ = AX, X(0) = I.

_Proof_. The formal term-by-term differentiation (6) shows that we need only to establish the uniform and absolute convergence in each interval  . Define

Note that   for all _i_ , _j_. For example, if

then | _A_ | = 14 + 14 = 28 > 5 + 10. If _B_ is another _n_ × _n_ matrix,

Therefore,  , and for every _i_ , _j_ , the _i_ , _j_ component of the matrix ( _k_!)−1 _A_ _k_ _t_ _k_ has absolute value  , where _α_ = | _A_ |. Also, therefore, the _i_ , _j_ component series in (7) is dominated by  . This completes the proof.

This argument made use of a particular _matrix norm_ | _A_ |. We will discuss matrix norms at length in Section 6.9.

#### **PROBLEMS**

**1.** Let _A_ be an _n_ × _n_ matrix. How shall you define the matrix functions cos _At_ and sin _At_?

**2.** If the power series ∑ _α_ _k_ _ζ_ _k_ converges (and therefore converges absolutely) for all _ζ_ , and if _A_ is an _n_ × _n_ matrix, prove the "convergence" of the series ∑ _α_ _k_ _A_ _k_.

**3.** Let _A_ be an _n_ × _n_ constant matrix, with det _A_ ≠ 0. Solve the initial-value problem

**4.** Let _A_ be a matrix which can be written in the _canonical form A_ = _B_ ∧ _B_ −1, where ∧ = diag (λ1, . . . , λ _n_ ). (In Chapter 4 we shall prove that most matrices have this property.) Let ∑ _α_ _k_ _ζ_ _k_ converge for | _ζ_ | < _γ_ and diverge | _ζ_ | > _γ_. Prove that _A_ _k_ = _B_ ∧ _k_ _B_ −1 for all _k_ = 0, 1, . . . . Under what condition on λ1, . . . , λ _n_ does the infinite series of matrices ∑ _α_ _k_ _A_ _k_ converge?

### **3.5 SOLUTION BY EIGENVALUES AND EIGENVECTORS**

For the general initial-value problem _x_ ′( _t_ ) = _Ax_ ( _t_ ) we have found the solution _x_ ( _t_ ) = _X_ ( _t_ ) _x_ (0) , where _X_ ( _t_ ) = exp ( _At_ ). This form of the solution tells us very little. For example, it does not tell us whether the solution tends to zero, oscillates, or becomes unbounded as _t_ → _∞_.

Let us look for particular solutions of _x_ ′( _t_ ) = _Ax_ ( _t_ ) of the form

The differential equation yields

Dividing both sides by _e_ λ _t_ gives

The _characteristic equation_ (2) should be interpreted by the question: _For which real or complex numbers λ does the homogeneous equation_ (2) _have a solution_ c ≠ 0?

We have already answered this question in Section 2.4. _The necessary and sufficient condition on λ which allows_ (2) _to have a solution_ c ≠ 0 _is_

**E XAMPLE 1.** Let

Equation (3) becomes

The roots of this equation are the _eigenvalues_

The _eigenvector c_ 1 belonging to λ1 is found from equation (2):

This homogeneous equation has the nonzero solution

Similarly, the equation 0 = (λ2 _I_ − _A_ ) _c_ 2 is solved by the eigenvector _c_ 2 = col(1,1 − _i_ ):

From the eigenvalues and eigenvectors, we construct the particular solutions _c_ exp λ _t_. These are the _eigensolutions:_

of the differential equation _x_ ′( _t_ ) = _Ax_ ( _t_ ). We now require the initial condition

Neither of the two eigensolutions (9) satisfies the required initial condition (10). Therefore, we try to find _x_ ( _t_ ) as a linear combination:

For every _α_ 1 and _α_ 2, the linear combination (11) solves the linear, homogeneous differential equation _x_ ′ = _Ax_. We _pick α_ 1 and _α_ 2 to satisfy the initial condition (10):

Observe that the eigenvectors _c_ 1 and _c_ 2 are linearly independent, and are hence a basis for _E_ 2. We can therefore solve for _α_ 1 and _α_ 2:

The required solution _x_ now takes the form

or, in real form,

The expression of _x_ ( _t_ ) as a linear combination of eigensolutions _c_ _j_ exp (λ _j_ _t_ ) tells us a great deal about the solution. Let λ _j_ = _γ_ _j_ \+ _iω_ _j_. Then

_The amplitude of the eigensolution_ xj(t) _is growing, staying constant, or decaying depending on whether γ_ _j_ > 0, _γ_ _j_ = 0, or _γ_ _j_ < 0. _The imaginary part ω_ _j_ _gives the angular frequency of the eigensolution oscillation_. In (13) of Example 1, there is a decay of amplitude because _γ_ 1 = _γ_ 2 = −3. There are oscillations with angular frequencies _ω_ 1 = 4, _ω_ 2 = −4.

In the general case of an _n_ × _n_ matrix, there are _n_ eigenvalues λ1, . . ., λ _n_ , because, as we shall prove in Section 4.1, the characteristic determinant det (λ _I_ − _A_ ) is a polynomial of degree _n_. There are corresponding eigenvectors _c_ 1, . . ., _c_ _n_. We shall see, in the next chapter, that the eigenvectors are usually (but not _always_ ) linearly independent and, therefore, are a basis for _E_ _n_. Given an initial vector, _x_ (0) , we can find a representation

Then the solution of the differential equation _x_ ′( _t_ ) = _Ax_ ( _t_ ) with initial state _x_ ( _0_ ), is a linear combination of eigensolutions:

The growth and the oscillation of the solution are observed from the real and imaginary parts of the eigenvalues.

Even when the eigenvectors _c_ 1, . . ., _c_ _n_ are dependent, a solution _x_ ( _t_ ) is provided by the Jordan canonical form, which is the subject of Chapter 5.

**E XAMPLE 2.** Let us obtain the fundamental solution _X_ ( _t_ ) of the differential equation

The fundamental solution _X_ is characterized by the initial-value problem

Let _Y_ ( _t_ ) be the matrix with columns that are the eigensolutions _x_ (1)( _t_ ), _x_ (2)( _t_ ), which were constructed in (9). Thus,

This matrix is not _X_ ( _t_ ) because it fails to satisfy the initial condition _X_ (0) = _I_. We now assert that

_Proof_. The matrix (21) equals _I_ when _t_ = 0. Further, since each column of _Y_ ( _t_ ) is a solution of _x_ ′ = _Ax_ , the whole matrix _Y_ satisfies _Y_ ′ = _AY_. Therefore,

Thus, the identity (21) is proved, and we may compute

where _Y_ ( _t_ ) is the matrix (20). From (22) we compute

_In general_ , _if_ c(1), . . . , c(n) _are independent eigenvectors belonging to_ λ1, . . . , λn, _the fundamental solution_ X(t) _solving_ (19) _is given by_

_where_ C _is the square matrix with columns of_ c(1), . . . , c(n).

_Proof_. The matrix of eigensolutions is given by

and _Y_ −1(0) = _C_ −1.

**E XAMPLE 3.** We will solve the inhomogeneous initial-value problem

According to Section 3.3, we have

From (23) we compute

#### **PROBLEMS**

**1.** Solve the initial-value problem

First state the problem in vector-matrix form.

**2.** Find the fundamental matrix-solution _X_ ( _t_ ) for the system in Problem 1.

**3.** Solve the initial-value problem

Use the result of Problem 2.

**4.** Find _all_ the eigensolutions _c_ exp λ _t_ of the system

Are the methods of this section directly applicable to this system?

**5.** For the system in Problem 4 find a fundamental matrix-solution satisfying _X′_ = _AX_ , _X_ (0) = _I_. Is there a matrix _Y_ of independent eigensolutions? Is the method of Section 3.3 for solving _x_ ′ = _Ax_ \+ _f_ still valid?

## **4 EIGENVALUES, EIGENVECTORS, AND CANONICAL FORMS**

### **4.1 MATRICES WITH DISTINCT EIGENVALUES**

The preceding chapter may be regarded as an introduction to the study of eigenvalues, eigenvectors, and canonical forms. If _A_ is an _n_ × _n_ matrix, we define the _eigenvalues_ of _A to be those numbers_ λ for which the _characteristic equation Ac_ = λ _c_ has a solution _c_ ≠ 0; the vector _c_ is called an _eigenvector_ belonging to the eigenvalue λ.

**Theorem 1**. _The number_ λ _is an eigenvalue of the square matrix_ A _if and only if_ det (λI − A) = 0.

_Proof_. This result follows from Theorem 1 of Section 2.4 if _A_ is replaced by λ _I_ − _A_.

For application to differential equations it is important that the _n_ × _n_ matrix _A_ have _n_ linearly independent eigenvectors. Unfortunately, this is not always true.

**E XAMPLE 1.** Let

Then

Therefore, λ = 7 is the only eigenvalue. The characteristic equation becomes

Thus, the components of _c_ must satisfy the equations

Therefore, we must have _c_ 2 = 0, and all eigenvectors of _A_ have the form

Every two vectors of this form are dependent. The differential equation _x_ ′ = _Ax_ is, of course, still solvable by means of the exponential matrix, c.f. Section 3.4, and we shall later find another solution by means of the Jordan canonical form.

**E XAMPLE 2.** Even if an _n_ × _n_ matrix does not have _n_ distinct eigenvalues, it _may_ have _n_ linearly independent eigenvectors. Let

then det (λ _I_ − _A_ ) = (λ − 1)2. The only eigenvalue is λ = 1. The characteristic equation becomes

Therefore, every nonzero _c_ is an eigenvector. We may, for example, choose

as two linearly independent eigenvectors.

**Theorem 2.** _Let A be an_ n × n _matrix. Then the characteristic determinant_. det (λI − A) _is a polynomial of degree_ n _in_ λ, _where the coefficient of_ λn _equals_ 1.

_Proof_. Let _B_ = λ _I_ − _A_. In the expansion

the only permutation _j_ which yields the power λ _n_ is the identity permutation. The other permutations _j_ give lower powers of λ. But

Therefore, the coefficient of λ _n_ equals 1.

It is conventional to say that every _n_ × _n_ matrix _A_ has _n_ eigenvalues, namely the numbers λ1, λ2, . . . , λ _n_ which appear in the factorization

Some of the numbers λ _j_ may be identical. Thus, the three eigenvalues of the 3 × 3 identity matrix are 1, 1, and 1. Alternatively, we may factor det (λ _I_ − _A_ ) in the form

where λ1, . . . , λ _r_ are different, and where the positive integers _m_ 1, . . . , _m_ _r_ are the _multiplicities of the eigenvalues_. Of course, _m_ 1 \+ . . . + _m_ _r_ = _n_. Thus, λ1 = 1 is an eigenvalue of multiplicity _m_ 1 = 3 of the 3 × 3 identity matrix. A matrix _A_ is said to have _distinct eigenvalues_ if all λ _j_ are different in the factorization (2) or, equivalently, if all _m_ _j_ = 1 and _r_ = _n_ in the factorization (3).

**Theorem 3**. _Let_ A _be an_ n × n _matrix with distinct eigenvalues_ λ1, . . . , λn. _Then_ A _has, correspondingly,_ n _linearly independent eigenvectors_ c1, . . . , cn. _Moreover, the eigenvector_ cj _belonging to the eigenvalue_ λj _is unique apart from a nonzero scalar multiplier_.

Thus, if _A_ has distinct eigenvalues, the differential equation _x_ ′ == _Ax_ with given initial state _x_ (0), has the solution given in Section 3.5 in terms of eigenvalues and eigenvectors.

_Proof of the Theorem_. Let _c_ 1 , . . . , _c_ _n_ be eigenvectors:

Suppose that _c_ 1 , . . . , _c_ _n_ are dependent. Let   be the _least_ positive integer such that _k_ of the _c_ 's are dependent. Without loss of generality, suppose that _c_ 1, . . . , _c_ _k_ are dependent:

In the last equation  , since all _c_ _j_ are nonzero. Furthermore, _all α_ _j_ are nonzero; otherwise, _k_ − 1 of the _c_ 's would be dependent, and _k_ would not be the least integer such that _k_ of the _c_ 's were dependent.

To eliminate _α_ _k_ , multiply (5) by the matrix _A_ − λ _k_ _I_. Now

Therefore, multiplication of (5) by _A_ − λ _k_ _I_ gives

Because the λ _j_ are distinct, λ1 − λ _k_ ≠ 0, . . . , λ _k_ −1 − ≠ 0. But all _α_ _j_ are nonzero. Therefore, all coefficients _α_ _j_ (λ _j_ − λ _k_ ) are nonzero in (7). Therefore, just _k_ −1 of the _c_ 's are dependent. This contradiction proves that _c_ 1 , . . . , _c_ _n_ are independent.

Now we prove the uniqueness. We must show that

Since the _n_ vectors _c_ 1, . . . , _c_ _n_ are independent, they are a basis for _E_ _n_. Therefore, there is a representation

Multiplication of (9) by _A_ − λ _j_ _I_ gives

But λ _i_ − λ _j_ ≠ 0 if _i_ ≠ _j_. Because _c_ 1, . . . , _c_ _n_ are independent, (10) implies _β_ _i_ = 0 if _i_ ≠ _j_. Equation (9) now takes the form _c_ = _β_ _j_ _c_ _j_ , so (8) is true with _μ_ = _β_ _j_.

#### **PROBLEMS**

**1.** Find independent eigenvectors for the matrix

Express the vector _x_ = col (1, 2) as a linear combination of the eigenvectors of _A_.

**2.** Does the singular matrix

have two independent eigenvectors?

**3.** Let

Can _x_ be an eigenvector of this matrix, _A_ , if _x_ 1 = 0? If λ is an eigenvalue of _A_ , and if _x_ 1 = 1, what is the form of the eigenvector _x_? What is the characteristic polynomial of _A_? Under what condition does _A_ have four linearly independent eigenvectors?

**4.** Generalize the last problem, replacing 4 by _n_. (The matrix, _A_ , is known as the _companion matrix_ of the polynomial λ _n_ \+ _α_ 1λ _n_ − 1 \+ . . . + _α_ _n_.)

### **4.2 THE CANONICAL DIAGONAL FORM**

Two _n_ × _n_ matrices _A_ and _B_ are called _similar_ if there exists an _n_ × _n_ matrix _T_ with det _T_ ≠ 0 for which

We will show that _similar matrices represent the same linear transformation in different coordinate systems_.

We think of _A_ as a linear transformation on the Euclidean space _E_ _n_. Each vector _x_ is mapped into a vector _x_ ′ by the transformation _x_ ′ = _Ax_. Let the vectors _t_ 1, . . . , _t_ _n_ be a basis for _E_ _n_. Then _x_ and _x_ ′ have representations

We may think of the numbers _y_ 1, . . . , _y_ _n_ as _coordinates_ of the vector _x_ with respect to the basis _t_ 1, . . . , _t_ _n_. The numbers _y_ ′1, . . . , _y_ ′ _n_ are the coordinates of _x_ ′. The transformation _x_ → _x_ ′ is equivalent to a transformation _y_ → _y_ ′. The coordinates _x_ _i_ , _x_ ′ _i_ refer to the basis _e_ 1, . . . , _e_ _n_ ; the coordinates _y_ _i_ , _y_ ′ _i_ refer to the basis _t_ 1, . . . , _t_ _n_.

If _x_ ′ = _Ax_ , what is the relationship between _y_ and _y_ ′? Let _T_ be the matrix whose columns are _t_ 1, . . . , _t_ _n_. We have det _T_ ≠ 0 because the _t_ 's form a basis. Equation (2) states

Now the transformation _x_ ' = _Ax_ has the representation

Solving for _y_ ′, we find _y_ ′ = ( _T_ −1 _AT_ ) _y_.

**E XAMPLE 1.** Let

The transformation _x_ ′ = _Ax_ is a rotation of _E_ 2 counterclockwise through 90°. Choose the basis

Now _x_ and _x_ ′ have the coordinates _y_ 1, _y_ 2 and _y_ ′1, _y_ ′2 given by the equations

In other words, _x_ = _Ty_ and _x_ ′ = _Ty_ ′ where

Now _x_ ′ = _Ax_ becomes _y_ ′ = _T_ −1 _ATy_ :

Multiplying the three matrices, we find _y_ ′ = _By_ with

This matrix _B_ represents the counterclockwise rotation of _E_ 2 through 90° _if_ the vectors in _E_ 2 are expressed in the coordinate system belonging to the basis (5).

**Theorem 1.** _Similar matrices have the same eigenvalues with the same multiplicities_.

_Proof_. Let _T_ −1 _AT_ = _B_. Now for all complex λ,

But

Therefore, det (λ _I_ − _B_ ) = det (λ _I_ − _A_ ); this completes the proof.

**E XAMPLE 2.** The matrices _A_ and _B_ in (4) and (6) both have the characteristic determinant λ2 \+ 1.

We may ask whether, conversely, two matrices must be similar if they have the same eigenvalues with the same multiplicities. The answer is no.

**E XAMPLE 3.** The matrices

cannot be similar because _T_ −1 _AT_ = _T_ −1 _T_ = _I_ for all _T_.

**Theorem 2.** _An_ n × n _matrix_ A _is similar to a diagonal matrix if and only if_ A _has_ n _linearly independent eigenvectors. In particular,_ A _is similar to a diagonal matrix if_ A _has distinct eigenvalues_.

This remarkable theorem states that practically every linear transformation, looked at in the right coordinate system, can be represented by uncoupled stretchings of the coordinates. However, the "right" coordinates and the "stretching" factors may have to be complex numbers. A diagonal matrix, ∧, to which _A_ is similar is known as a _canonical diagonal form_ related to _A_.

_Proof of the Theorem_. First suppose that _A_ is similar to a diagonal matrix:

Let _t_ 1 , . . . , _t_ _n_ be the linearly independent columns of the nonsingular matrix _T_. Multiply both sides of (8) on the left by _T_. The _j_ th column of _AT_ is _At_ _j_ ; the _j_ th column of _T∧_ is _α_ _j_ _t_ _j_. Therefore, (8) implies

Thus, the independent vectors _t_ _j_ are eigenvectors of _A_. By Theorem 1, _the numbers α_ _j_ _are the eigenvalues of_ A _with the same multiplicities_.

Conversely, suppose that _A_ has _n_ independent eigenvectors _t_ 1 , . . . , _t_ _n_. Let _T_ be the nonsingular matrix with columns that are the vectors _t_ _j_. From the characteristic equations (9) we conclude that _AT_ = _T∧_ , and hence _T_ −1 _AT_ = ∧.

**E XAMPLE 4.** Let

The eigenvalues of _A_ are _α_ 1 = _i_ and _α_ 2 = − _i_. Corresponding eigenvectors are

Now

If we represent any vector _x_ by coordinates _y_ 1, _y_ 2 with respect to the basis _t_ 1, _t_ 2, we have

The eigenvalues _i_ and − _i_ give uncoupled "stretchings" of the coordinates _y_ 1, _y_ 2.

**E XAMPLE 5.** Let _A_ be an _n_ × _n_ matrix with _n_ linearly independent eigenvectors _c_ 1 , . . . , _c_ _n_. Let _C_ have the vectors _c_ _j_ as columns. To solve the initial-value problem

make the change of coordinates _x_ = _Cy_. Now (12) becomes

Multiplication of (13) on the left by _C_ −1 gives

where _C_ −1 _AC_ = ∧ = diag (λ1, . . . , λ _n_ ). But the system (14) is _uncoupled_ ; by components it reads

Therefore, we have the solution

The _uncoupling_ property of the diagonal form is used to solve many sorts of linear equations.

**E XAMPLE 6.** Consider the coupled difference equations

Let _x_ ( _k_ ) = col ( _u_ _k_ , _v_ _k_ ). Then (17) takes the form

To uncouple this system, we introduce the canonical diagonal form of _A_. From Example 1, Section 3.5, we have

and ∧ = diag (−3 + 4 _i_ , −3 − 4 _i_ ). Make the change of coordinates

Now (18) becomes _y_ _k_ +1 = ∧ _y_ _k_ , and this system is uncoupled. _If y_ _k_ = col ( _r_ _k_ , _s_ _k_ ), then

These equations are readily solvable:

To find _u_ _k_ and _v_ _k_ , we write out equation (20):

Therefore, for _k_ = 0, 1, . . . ,

The solutions _u_ _k_ , _v_ _k_ are completely determined by given initial values _u_ 0, _s_ 0. By (23), the initial values _u_ 0, _s_ 0 determine the numbers _r_ 0, _s_ 0 by the equations

#### **PROBLEMS**

**1.** Solve the system of difference equations in Example 6 under the initial conditions: _u_ 0 = 1, _v_ 0 = 2. Verify that the solution given by (23) is real valued even though it is expressed in terms of complex numbers.

**2.** Solve the difference equation _f_ _n_ +1 = _f_ _n_ \+ _f_ _n_ −1 for the _Fibonacci numbers_ by obtaining a first-order system _x_ ( _n_ +1) = _Ax_ ( _n_ ) for the vectors

Assume _f_ 0 = _f_ 1 = 1.

**3.** Let _A_ have independent eigenvectors _u_ 1, . . . , _u_ _n_. Let _B_ = _T_ −1 _AT_. Does _B_ have _n_ independent eigenvectors _v_ 1, . . . , _v_ _n_? How are the eigen-vectors of similar matrices related?

**4.** Using the technique of Example 5, solve the system of differential equations _dx_ / _dt_ = _Ax_ , where _A_ is the matrix in formula (18). Assume that _x_ (0) = col (1,2).

**5.** If _A_ is similar to _B_ , show that the matrices _A_ _k_ and _B_ _k_ are similar for all powers _k_ = 0, 1, 2, . . . . If det _A_ ≠ 0, also show that _A_ −1 is similar to _B_ −1.

**6.** If _A_ is similar to _B_ , and if _B_ is similar to _C_ , show that _A_ is similar to _C_.

**7.** * Let _A_ be an _n_ × _n_ matrix with distinct eigenvalues λ _j_. Let _x_ (0), _x_ (1), . . . solve the difference equation _x_ ( _k_ +1) = _Ax_ ( _k_ ). Under what condition on the eigenvalues λ _j_ can we be sure that all components of the vector _x_ ( _k_ ) tend to zero as _k_ → ∞?

**8.** * Let _A_ be an _n_ × _n_ matrix with distinct eigenvalues λ _j_. Let _x_ ( _t_ ) solve the differential equation _dx_ / _dt_ = _Ax_. Under what condition on the eigenvalues λ _j_ can we be sure that all components of the vector _x_ ( _t_ ) tend to zero as _t_ → ∞?

### **4.3 THE TRACE AND OTHER INVARIANTS**

The sum of the eigenvalues of an _n_ × _n_ matrix _A_ , is called its _trace:_ tr _A_ = ∑ λ _j_. This number is minus the coefficient of λ _n_ −1 in the expansion

If matrices _A_ and _B_ are similar, they have the same trace because they have the same characteristic polynomial. In this sense, the trace is _invariant_ in similarity transformations _B_ = _T_ −1 _AT_.

Another invariant is the determinant. Setting λ = 0 in (1) gives det (− _A_ ) = (− λ1) . . . (− λ _n_ ). But det (− _A_ ) = (−) _n_ det _A_. Therefore,

Evidently, all of the coefficients _c_ 1, _c_ 2, . . . , _c_ _n_ are invariant in similarity transformations.

**Theorem 1.** _Let_ A _be an_ n × n _matrix_. _Then_

_Proof_. In the expansion (1) we must show that

Let _B_ = λ _I_ − _A_. In the expansion

we shall find the coefficient _c_ 1 of λ _n_ −1. The only permutation _j_ which contributes _n_ − 1 factors λ is the permutation _j_ 1 = l, _j_ 2 = 2, . . . , _j_ _n_ = _n_. If _j_ _i_ ≠ _i_ for one _i_ , then _j_ _i_ ≠ _i_ for at least two _i_ , which implies that ∏ _b_ _ij_ _i_ has degree   in λ. Hence, _c_ 1 is the coefficient of λ _n_ −1 in the single term

Therefore, _c_ 1 = − _a_ 11 − . . . − _a_ _nn_. But the identity (1) yields _c_ 1 = −λ1, − . . . −λ _n_. This completes the proof.

**E XAMPLE 1.** Let

Then the sum of the eigenvalues of _A_ must equal 5 + 3 = 8

_All_ coefficients _c_ 1, _c_ 2, . . . , _c_ _n_ of the characteristic polynomial are invariant under similarity transformations. This was proved in Theorem 1 of the last section. We have shown that _c_ 1 = − ∑ _a_ _ii_ and _c_ _n_ = (−) _n_ det _A_. These results can be generalized.

**Theorem 2.** _Let_  . _Let_ A _be an_ n × n _matrix_ , _with characteristic determinant_ (1). _Then_

_where_ ∆(i1, . . . , im) _is the_ m × m _determinant formed from rows_ i = i1, i2, . . . , im _and columns_ j = i1, i2, . . . , im.

**E XAMPLE 2.** Let

For _m_ = 2 formula (7) says

_Proof of the Theorem_. We have

To find the coefficient _c_ _m_ of λ _n_ − _m_ , we must take _δ_ _ij_ 1λ from _n_ − _m_ of the factors (λ _δ_ − _a_ ) and take − _a_ _ij_ _i_ from the remaining _m_ factors in all possible ways. Thus,

The only nonzero terms come from permutations _j_ for which the product of Kronecker deltas is nonzero, i.e., for which _i_ = _j_ _i_ when _i_ ≠ _i_ 1, _i_ 2, . . . , _i_ _m_. Now (8) becomes

where _k_ runs over all permutations of _i_ 1, . . . , _i_ _m_. Thus,

This is the required identity (7). Note that _S_ ( _k_ ) in formula (9) equals _S_ ( _j_ ) in formula (8) when _j_ _i_ = _i_ for all _i_ ≠ _i_ 1, . . . , _i_ _m_ , since any sequence of _t_ interchanges that reduces the permutation _k_ to the natural order _i_ 1, . . . , _i_ _m_ , simultaneously reduces _j_ to the natural order 1, . . . , _n_.

#### **PROBLEMS**

**1.** Compute λ1λ2 \+ λ1λ3 \+ λ1λ4 \+ λ2λ3 \+ λ2λ4 \+ λ3λ4 for the matrix

**2.** For the matrix in Problem 1, compute λ1 \+ λ2 \+ λ3 \+ λ4. Now, using the result of Problem 1, compute   \+   \+   \+  .

**3.** Show that tr ( _AB_ ) = tr ( _BA_ ) even if _AB_ ≠ _BA_.

**4.** Suppose det _A_ = 0. Show that, for all sufficiently small | _ε_ | > 0, det ( _A_ \+ _εI_ ) ≠ 0.

**5.** Prove that det (λ _I_ − _AB_ ) ≡ det (λ _I_ − _BA_ ) for all λ. Hence, show that all the invariants _c_ 1 , . . . , _c_ _n_ are the same for _AB_ and for _BA_. (First prove the result by assuming det _A_ ≠ 0. If det _A_ = 0, obtain the required identity by replacing _A_ by _A_ \+ _εI_ and by taking the limit as _ε_ → 0.)

### **4.4 UNITARY MATRICES**

The study of invariance under transformations leads to many interesting questions. For example, we may ask: _Which_ n × n _matrices_ U _transform vectors_ x _into vectors_ y _with the same length_? If _x_ has real components, we mean by _length_ the Euclidean norm

If _x_ has complex components, we must replace this definition by

Otherwise certain nonzero vectors, e.g., _x_ = col (1, _i_ ), could have zero length. We ask: For which _n_ × _n_ matrices _U_ is

**E XAMPLE 1.** For _n_ = 2 and real _x_ the rotation matrices

have the property (3). They also satisfy (3) for complex _x_ ; since

**E XAMPLE 2.** For all _x_ the matrix

preserves length:

Before answering the question let us discuss the _n_ -dimensional length (2). Can we define an "angle" _ϕ_ between two _n_ -dimensional vectors _x_ and _y_? For _n_ = 2 and 3 we have, for real _x_ and _y_ ,

The numerator of this fraction is the _inner product_ ( _x_ , _y_ ) defined as

If the components _x_ _j_ , _y_ _j_ are allowed to be complex numbers, the inner product is defined as

The inner product has several important elementary properties:

For _n_ > 3, even for real _x_ and _y_ , it is not obvious that the "cosine" (6) has absolute value  .

**Theorem 1.** ( _The Cauchy-Schwarz Inequality_ )

_with equality if and only if_ x _and_ y _are dependent_.

_Proof_. If _x_ = 0 or _y_ = 0 then both sides of (9) are equal to zero. Therefore, suppose _x_ ≠ 0 and _y_ ≠ 0. Now (9) takes the form

or

where _a_ = _x_ /|| _x_ || and _b_ = _y_ /|| _y_ ||:

Evidently,

with equality if and only if the argument is constant:

where the argument of zero may be chosen at will. Next,

with equality if and only if | _a_ _v_ | = | _b_ _v_ |. But _a_ = _x_ /|| _x_ || and _b_ = _y_ /|| _y_ || are unit vectors! Therefore, summation of (15) for _v_ = 1, . . . , _n_ gives

This establishes the required inequality (11), equivalent to (9), with equality if and only if, for _v_ = 1, . . . , _n_ ,

If | _x_ _v_ | = _r_ _v_ , arg _x_ _v_ = _θ_ _v_ , and λ = || _y_ ||/|| _x_ ||, formula (17) states

Thus, (9) holds, with equality if and only if _y_ equals λ exp (− _iψ_ ) times _x_. This completes the proof.

**Lemma 1.** _An_ n × n _matrix_ U _preserves lengths if and only if it preserves inner products_ , _i.e._ ,

_if and only if_

_Proof_. The identity (20) implies (19) as follows: For all _x_ , define _y_ = _x_. Then (20) states

which is equivalent to (19).

To show that (19) implies (20) is harder. The inner product can be related to the length by the identity

or

To obtain Im ( _x_ , _y_ ) from the last identity, replace the vector _y_ by the vector _iy_ :

But ( _x_ , _iy_ ) = − _i_ ( _x_ , _y_ ), so Re ( _x_ , _iy_ ) = Im ( _x_ , _y_ ). Therefore,

If we replace _x_ by _Ux_ , and _y_ by _Uy_ , in the preceding identities, we find

But the preservation of _all_ lengths (19) implies

Now the identities (21′), (22′) become

From (21) and (22) we now conclude

which is the required result (20).

Since a length-preserving matrix _U_ preserves inner products,

where _e_ 1, _e_ 2, . . . , _e_ _n_ are the columns of the identity matrix. But _Ue_ _j_ and _Ue_ _k_ are simply the _j_ th and _k_ th columns of _U_. Formula (23) shows that _the columns of a length-preserving matrix are mutually orthogonal unit vectors_. Is the converse true? Given (23), we have for all _x_

This establishes the converse. We have proved

**Theorem 2.** _The_ n × n _matrix_ U _preserves Euclidean length_ || x || = (∑ | x _v_ |2)1/2 _if and only if the columns_ uj _of_ U _are mutually orthogonal unit vectors_ : (uJ, uk) = _δ_ jk.

For any matrix _A_ we define the _adjoint matrix A_ * to be the complex conjugate of the transpose of _A_. For example,

In particular, for column-matrices (vectors) _x_ and _y_ , we have

The orthogonality and unit length of the columns of _U_ can now be summarized in the formula

In fact, by the rule of matrix multiplication, the _j_ , _k_ component of _U_ * _U_ is simply the sum _u_ _j_ * _u_ _k_ = ( _u_ _k_ , _u_ _j_ ), which is required by (25) to equal _δ_ _jk_ = _δ_ _kj_.

Square matrices _U_ for which _U_ * _U_ = _I_ are called _unitary_.

**Theorem 3.** _If_ U _and_ V _are_ n × n _unitary matrices_ , _so are_ U* _and_ UV.

_Proof. UV_ is unitary because it preserves length:

_U_ * is unitary because the given equation _U_ * _U_ = _I_ characterizes _U_ * as the inverse of _U_. Since a left-inverse is a right-inverse, we have _UU_ * = _I_. But _U_ = ( _U_ *)*, as we see by taking complex conjugates and taking the transpose twice. Therefore, ( _U_ *)*( _U_ *) = _I_.

This theorem has the amusing corollary that the columns of a square matrix are mutually orthogonal unit vectors if and only if the _rows_ are mutually orthogonal unit vectors.

#### **PROBLEMS**

**1.** Let _A_ be an _m_ × _n_ matrix. Show that ( _Ax_ , _y_ ) = ( _x_ , _By_ ) for all _n_ -component vectors _x_ and all _m_ -component vectors _y_ if and only if  .

**2.** If _A_ −1 exists, show that ( _A_ *)−1 = ( _A_ −1)*.

**3.** Show that ( _AB_ )* = _B_ * _A_ *.

**4.** Show that all the eigenvalues of a unitary matrix are numbers λ = _e_ _iθ_ on the unit circle.

**5.** Assume that _ρ_ 1 > 0, . . . , _ρ_ _n_ > 0. Redefine the "length" of a vector as

Which matrices _V_ preserve this "length"?

**6.** Let _U_ be unitary, and let _B_ = _UA_ and _C_ = _AU_. Show that

where the trace is the sum of the diagonal elements.

**7.** * Define the Lorentz-metric _x_ 2 − _c_ 2 _t_ 2 from the special theory of relativity. We ask for linear transformations _x_ ′ = _αx_ \+ _βt_ , _t′_ = _γx_ \+ _δt_ such that _x_ ′2 − _c_ 2 _t_ ′2 ≡ _x_ 2 − _c_ 2 _t_ 2 for all _x_ and _t_ , where _a_ > 0 and _δ_ > 0. Show that all such linear transformations have the form

**8.** * Show that rigid motions are necessarily _linear_ transformations: For _i_ = 1, . . . , _n_ let _y_ _i_ = _f_ _i_ ( _x_ 1, . . . , _x_ _n_ ) be real-valued, twice continuously differentiable functions of the real vector _x_ = ( _x_ 1, . . . , _x_ _n_ ). For all vectors _x_ and all increments Δ _x_ suppose that

Then prove that the functions _f_ _i_ ( _x_ ) have the form

where the _c_ _i_ and the _u_ _ij_ are _constants_ , with ∑( _i_ ) _u_ _ij_ _u_ _ik_ = _δ_ _jk_. (Method: Expand each _f_ _i_ ( _x_ \+ Δ _x_ ) in a Taylor series with a remainder involving the second derivatives. Letting Δ _x_ → 0, show that the second derivatives are all zero.)

### **4.5 THE GRAM-SCHMIDT ORTHOGONALIZATION PROCESS**

In this section we will show that every finite-dimensional linear vector space with real or complex scalars has a basis consisting of mutually orthogonal unit vectors. In the following theorem we will do somewhat more. Suppose that we are given linearly independent vectors _a_ 1, _a_ 2, . . . , _a_ _m_. We will show how to orthogonalize these vectors successively. First we will find a unit vector _u_ 1 which is a multiple of _a_ 1. Then we will find a unit vector _u_ 2, orthogonal to _u_ 1, such that _u_ 1 and _u_ 2 span the same plane as _a_ 1 and _a_ 2. Then we construct a unit vector _u_ 3, orthogonal to _u_ 1 and _u_ 2, such that _u_ 1, _u_ 2, _u_ 3 span the same linear space as _a_ 1, _a_ 2, _a_ 3. Finally we shall have mutually orthogonal unit vectors _u_ 1, . . . , _u_ _m_ spanning the same linear space as _a_ 1, . . . , _a_ _m_.

The Gram-Schmidt process has many applications. For example, it is used to construct the orthogonal polynomials and other orthogonal sets of functions used in mathematical physics. Because of these applications, we will be rather general in our definitions of "inner product" and "orthogonality."

Consider a linear vector space, _L_ , defined over a field of scalars, _F_. For us, _F_ will be either the field of real numbers or the field of complex numbers. We suppose that for every two vectors _x_ and _y_ a complex or real number ( _x_ , _y_ ) is defined, and that the following laws hold:

In this case, the functional ( _x_ , _y_ ) is called an _inner product_. Two vectors, _x_ and _y_ , are called _orthogonal_ if ( _x_ , _y_ ) = 0. The vector _u_ is called a _unit vector_ if ( _u, u_ ) = 1.

**E XAMPLE 1.** Let _L_ be the space of vectors with _n_ real components, defined over the field, _F_ , of real numbers. Then we may define

**E XAMPLE 2.** Let _L_ be the space of vectors with two complex components, defined over the field, _F_ , of complex numbers. Then we may define

Note that

with ( _x, x_ ) = 0 only when _x_ = 0.

**E XAMPLE 3.** Let _L_ be the linear space of continuous, complex-valued functions _x_ = _ϕ_ ( _t_ ) defined for  . Let _F_ be the field of complex numbers. Then we may define the inner product

In this space the "vectors"

are "orthogonal."

**Theorem 1.** _Let_ L _be a linear vector space defined over the field_ , F, _of real numbers or of complex numbers. Let an inner product_ (x, y) _be defined which obeys the laws_ (1). _Let_ a1, a2, . . . , am _be linearly independent vectors in_ L. _Then there are mutually orthogonal unit vectors_ u1, u2, . . . , um _such that, for each_ j = 1, 2, . . . , m,

_Proof_. Since _a_ 1 ≠ 0, we have ( _a_ 1, _a_ 1) > 0. For any vector _x_ , define

Then we set

For any   suppose that we have constructed unit vectors _u_ 1, . . . , _u_ _k_ such that (2) holds for all  . If  , we will show how to construct _u_ _k_ +1 so that (2) holds for all  . Let _v_ be a vector of the form

Since _u_ 1, . . . , _u_ _k_ are linear combinations of _a_ 1, . . . , _a_ _k_ , we have

Therefore, _υ_ ≠ 0 because _a_ _k_ +1 _a_ l, . . . , _a_ _k_ are linearly independent. We now choose the coefficients _α_ 1, . . . , _α_ _k_ so that _υ_ is orthogonal to _u_ 1, _u_ 2, . . . , _u_ _k_. For _j_ ≤ _k_ , we have, from (4),

since ( _u_ _s_ , _u_ _j_ ) = _δ_ _sj_ ≡ 0 or 1, depending on whether _s_ ≠ _j_ or _s_ = _j_. Thus, the nonzero vector _υ_ is orthogonal to all the unit vectors _u_ _j_ ( _j_ ≤ _k_ ) if we define

To make _υ_ into a unit vector, we divide it by its length

The relations (2) hold now for all _j_ ≤ _k_ \+ 1, where ( _u_ _s_ _u_ _j_ ) = _δ_ _sj_. The theorem now follows by induction.

**Corollary.** _Every finite-dimensional linear vector space with an inner product has a basis consisting of orthogonal unit vectors u_ 1, . . . , _u_ _m_.

_Proof_. The desired result follows at once from the theorem if we let _a_ 1, . . . , _a_ _m_ be any basis for the space.

**E XAMPLE 4.** The vectors

are a basis for the plane _x_ 1 \+ _x_ 2 \+ _x_ 3 = 0 in three-dimensional, real Euclidean space, where we define the inner product to be

Using the Gram-Schmidt process, we first compute

To find _u_ 2, set

We compute

Thus, from (7),

Setting, _u_ 2= _υ_ ∕|| _υ_ ||, where  , we find

**E XAMPLE 5.** Consider the linear vector space of polynomials _x_ =  ( _t_ ) of degree ≤ _N_ defined over the field of real numbers. Introduce the inner product

We wish to obtain an orthogonal basis

We begin with the basis

We then compute

And then

Now write

We compute

Now

Hence,

To compute _u_ (2), we define the "vector"

Here

Thus

Hence,

and the process could be continued indefinitely. The polynomials _u_ (0), _u_ (1), _u_ (2), . . . are normalized _Laguerre_ polynomials.

#### **PROBLEMS**

**1.** The vectors

are a basis for the plane _x_ 1 \+ _x_ 2 \+ _x_ 3 \+ _x_ 4 = 0 in four-space. Using the Gram-Schmidt process, find a basis consisting of orthogonal unit vectors _u_ 1, _u_ 2, _u_ 3.

**2.** Consider the three-dimensional space spanned by the "vectors" _a_ (0) = 1, _a_ (1) = _t_ , _a_ (2)= _t_ 2 in the space of polynomials in _t_ of degree ≤ 100. Use real scalars. Define the inner product

Using the Gram-Schmidt process, find an orthogonal set of unit vectors _u_ (0), _u_ (l), _u_ (2) spanning the same space as _a_ (0), _a_ (1), _a_ (2).

**3.** Let _L_ be any _n_ -dimensional vector space defined over the field of complex numbers. Let _b_ (1), . . . , _b_ ( _n_ ) be any basis. If arbitrary vectors _x_ and _y_ have representations

show that the definition

has all the properties (1) required of an inner product. Find orthogonal unit vectors with respect to this inner product.

**4.** Let _a_ 1, . . . , _a_ _m_ be linearly independent vectors over the field of complex numbers. Define _α_ _ij_ = ( _a_ _i_ )* _a_ _j_ =( _a_ _j_ , _a_ _i_ ). Show that det ( _α_ _ij_ ) ≠ 0 by showing that an eigenvector _x_ of the matrix ( _α_ _ij_ ) belonging to the eigenvalue 0, would satisfy the relation || ∑ _x_ _i_ _a_ _i_ ||2 = 0.

**5.** * Define _a_ 1, . . . , _a_ _m_ and _α_ _ij_ as in Problem 4. Let _υ_ 1 = _a_ 1. Define the vectors _υ_ 2, . . . , _υ_ _n_ by the "determinants"

where we expand by the last row. For instance, _υ_ 2 = _α_ 11 _a_ 2— _α_ 12 _a_ l. Show that the vectors _u_ l, . . . , _u_ _m_ created by the Gram-Schmidt process are scalar multiples

Find the scalars _ρ_ 1 , . . . , _ρ_ _m_ in terms of ∆ _k_ = det ( _α_ _ij_ )( _i_ , _j_ =1, . . . , _k_ ). Note that ( _υ_ _k_ , _υ_ _k_ ) = ( _υ_ _k_ , ∆ _k_ -1 _a_ _k_ ) because _υ_ _k_ is orthogonal to _a_ 1, . . . , _a_ _k_ -1 and because the coefficient of _a_ _k_ is ∆ _k_ -1 in the expansion of _υ_ _k_ as a linear combination of _a_ 1, . . . , _a_ _k_.

### **4.6 PRINCIPAL AXES OF ELLIPSOIDS**

For dimension _n_ = 2 or _n_ = 3, an ellipse or an ellipsoid centered at the origin is represented by an equation

or, if we define _a_ _ij_ = _a_ _ji_ for _i_ > _j_ ,

Let _A_ = ( _a_ _ij_ )( _i_ , _j_ = 1, . . . , _n_ ). The real matrix _A_ is called _symmetric_ because _a_ _ij_ = _a_ _ji_ for all _i_ ≠ _j_. In terms of the inner product, the equation (1) takes the form

since _x_ has real components.

A _principal axis_ of the ellipsoid is a vector extending from the origin to a point _x_ on the ellipsoid such that the vector is normal to the ellipsoid at the point _x_. For _n_ = 2, there is an illustration in Figure 4.1.

**Figure 4.1**

An ellipse has two independent principal axes. An ellipsoid, in three dimensions, has three independent principal axes. From analytic geometry we know that _the principal axes are, or may be chosen to be, mutually orthogonal,_ and we shall prove and generalize this result in the next section. The cautious phrase "or may be chosen to be" refers to spheres or other ellipsoids of revolution, for which the choice of three mutually orthogonal principal axes is not unique.

We will now show that _the principal axes of an ellipsoid are the eigenvectors of the real, symmetric matrix_ A = ( _a_ _ij_ ). From calculus we know that a differential _dx_ = ( _dx_ 1 , . . . , _dx_ _n_ ) on a surface  ( _x_ 1 , . . . , _x_ _n_ ) = const, satisfies the relation

The vector grad   = ( _∂ _ ∕ _∂x_ _k_ )( _k_ = 1, . . . , _n_ ), being normal to all differentials _dx_ = ( _dx_ _k_ ) near the point _x_ , is therefore a vector normal to the surface _ϕ_ = const at the point _x_. Letting  ( _x_ ) be the _quadratic form_ Σ Σ _a_ _ij_ _x_ _i_ _x_ _j_ , we may compute the normal vector to the ellipsoid   = 1:

By the rule for differentiating a product,

Inserting (5) in (4) and summing, we find

Since _a_ _ik_ = _a_ _ki_ , we thus have

**Figure 4.2**

The normal vector _Ax_ and the radial vector _x_ are illustrated in Figure 4.2. The vector _x_ in the figure is clearly not a principal axis because it does not have the same direction as _Ax_. The equation defining a _principal axis x_ is

where ( _Ax_ , _x_ ) = 1. Since _x_ ≠ 0, _a principal axis is an eigenvector of_ A.

_The length of the principal axis associated with the eigenvalue_ λi _is_ .

To see this, take the inner product of both sides of (8) with _x_ :

But ( _Ax_ , _x_ ) = 1 for a point _x_ on the ellipsoid, and (λ _x_ , _x_ ) = λ|| _x_ ||2. Now (9) states 1 = λ|| _x_ ||2, which yields the length   Evidently, _a quadratic form_ (Ax, x) = 1 _representing an ellipsoid must come from a matrix with positive eigenvalues._

**E XAMPLE 1.** Consider the ellipse

This equation states that ( _Ax_ , _x_ ) = 1, where

The eigenvalues of _A_ are the roots of λ2 − 4λ + 2 = 0. Thus,

The principal axis _x_ (1) belonging to λl satisfies

Therefore, the first principal axis equals

Similarly, it is possible to compute the second principal axis

_Observe that the axes_ x(1) _and_ x(2) _are orthogonal_. The constants _α_ and _β_ could be determined so that _x_ (1) and _x_ (2) both lie on the ellipse (10). Further,

_The principal axes yield natural coordinates for an ellipsoid_ (Ax, x) = 1. Define mutually orthogonal unit vectors _u_ l, _u_ 2, . . . , _u_ _n_ by dividing the principal axes by their lengths. Let an arbitrary vector _x_ be represented as a linear combination

The numbers _y_ 1, _y_ 2, . . . are coordinates with respect to the orthogonal basis _u_ 1, _u_ 2, . . . . We now have

because the principal axes are eigenvectors of _A_. Since ( _u_ j, _u_ k) = _δ_ jk, the equation ( _Ax_ , _x_ ) = 1 becomes

or

or

In matrix notation, let _U_ be the unitary matrix with columns that are the normalized axes _u_ 1, . . . , _u_ _n_. Then (15) states _x_ = _Uy_. Now

But we readily verify that ( _z_ , _Uy_ ) = ( _U_ * _z_ , _y_ ) for any _z_ , by the definition of the inner product. Setting _z_ = _AUy_ , we have

But _AU_ = _U_ Λ, where Λ = diag (λ1 , . . . , λ _n_ ). Since _U_ is unitary,

Thus ( _Ax_ , _x_ ) = 1 becomes (Λ _y_ , _y_ ) = 1, a restatement of (19).

#### **PROBLEMS**

**1.** Consider the ellipse

If this equation is written in the form ( _Ax_ , _x_ ) = 1, what is the matrix _A_? What are the eigenvalues and eigenvectors of _A_? Find principal axes for the ellipse, and compute their lengths by the method of Example 1. If _U_ * _AU_ = Λ, where _U_ is unitary and Λ is diagonal, what are _U_ and Λ? What is the equation (19)?

**2.** Show that every _symmetric_ , real 2 × 2 matrix _A_ has real eigenvalues. Prove that the eigenvalues are both positive if and only if _a_ 11 > 0 and det _A_ > 0.

**3.** Answer all the questions in Problem 1 for the ellipse 2  − 2 _x_ 1 _x_ 2 \+   = 1.

### **4.7 HERMITIAN MATRICES**

If _A_ is an _n_ × _n_ matrix with real or complex components, we define the _adjoint matrix_ , _A_ *, to be the complex conjugate of the transpose of _A_. Thus

and

If _A_ _T_ is the transpose of _A_ , we have

If we define the _inner product_ of two vectors, _x_ and _y_ , to be

then the adjoint matrix satisfies the identity

because

Conversely, the identity

This follows from setting _x_ = _e_ _j_ and _y_ = _e_ _k_ , which yields _a_ _kj_ = _b_ _jk_.

A real, symmetric matrix _A_ is its own adjoint: _A_ = _A_ _T_ = _A_ *. _In general, Hermitian matrices are defined as self-adjoint matrices_ :

**E XAMPLE 1.** The matrices

are Hermitian, but the complex, symmetric matrix

is not Hermitian.

**Theorem 1.** _The eigenvalues of a Hermitian matrix are real_ , _and eigenvectors belonging to different eigenvalues are orthogonal_.

_Proof_. Suppose that _H_ = _H_ * and

Then ( _Hx_ , _x_ ) = (λ _x_ , _x_ ) = λ( _x_ , _x_ ), and λ is the _Rayleigh quotient_

The denominator ( _x_ , _x_ ) is positive. The numerator is real because it equals its own conjugate:

Therefore, λ is the quotient of two real numbers.

Next, suppose

Then

But ( _Hx_ , _y_ ) = ( _x_ , _Hy_ ), and therefore

But   because _μ_ is an eigenvalue of a Hermitian matrix! Since λ ≠ _μ_ by assumption, we have the orthogonality: ( _x_ , _y_ ) = 0.

The last result shows that, if _H_ = _H_ * has distinct eigenvalues, then it can be diagonalized by a unitary similarity transformation

In fact, we may take for the columns of _U_ a set of _n_ mutually orthogonal unit eigenvectors belonging to the _n_ different eigenvalues of _H_.

The remarkable fact is that _the canonical diagonalization_ (10) _of Hermitian matrices is_ always _possible_ , _even if the eigenvalues_ λ1, . . . , λ _n_ _are not distinct_. We shall prove this result in the next theorem, but the following heuristic argument shows why it is true: If the eigenvalues of _H_ are not distinct, think of _H_ as the limit of nearby Hermitian matrices _H_ _є_ with distinct eigenvalues. For every small   > 0 there are _n_ mutually perpendicular eigenvectors. As   → 0, by the perpendicularity, there is no way in which any two of the eigenvectors can coalesce in the limit; the eigenvectors are held rigidly apart at 90° angles.

**Theorem 2.** _Let_ H _be an_ n × n _Hermitian matrix_ , _with eigenvalues_ λ1, λ2 , . . . , λ _n_. _Then_ H _has_ n _mutually orthogonal unit eigenvectors_ u1, . . . , u _n_ , _with_

_where_ U _is the unitary matrix with columns_ u1, . . . , un _and_ Λ = diag (λ1, . . . ,λ _n_ ).

To prove this theorem, we need a preliminary result:

**Lemma 1.** _If any_ n × n _matrix_ A _maps a subspace_ Ld _of_ En(C) _into itself_ , _then_ A _has an eigenvector in_ Ld _if_ Ld _has dimension_ d ≥ 1.

_Proof_. If _L_ _d_ has dimension _d_ , let _b_ 1, . . . , _b_ _d_ be any basis for _L_ _d_. Then _Ab_ _j_ is a linear combination of _b_ 1, . . . , _b_ _d_ :

If _B_ is the _n_ × _d_ matrix with columns of _b_ 1, . . . , _b_ _d_ , we have _AB_ = _BM_ , where _M_ is the _d_ × _d_ matrix

Since the characteristic equation det (λ _I_ – _M_ ) = 0 has at least one root, _M_ has some eigenvector _y_ :

Now, because _AB_ = _BM_ , we have

The vector _x_ = _By_ is nonzero because _y_ ≠ 0 and because the columns of _B_ are independent. Therefore,

is an eigenvector of _A_ lying in the space _L_ _d_.

_Proof of Theorem 2_. The _n_ × _n_ matrix _H_ has a unit eigenvector, _u_ 1, belonging to some eigenvalue, λ1. Suppose that we have found _k_ mutually orthogonal unit eigenvectors _u_ 1, . . . , _u_ _k_ with 1 ≤ _k_ < _n_. Define _L_ _n_ – _k_ to be the linear space of vectors orthogonal to all the eigenvectors _u_ 1, . . . , _u_ _k_. Since _H_ is Hermitian, we can show that _H_ maps _L_ _n_ – _k_ into itself: If ( _x_ , _u_ _j_ ) = 0, where _Hu_ _j_ = λ _j_ _u_ _j_ , then

By the lemma, we can show that _H_ has an eigenvector in _L_ _n_ – _k_ if _L_ _n_ – _k_ contains any nonzero vector _x_. Since _k_ < _n_ , the vectors _u_ 1, . . . , _u_ _k_ cannot be a basis for the _n_ -dimensional space _E_ _n_ , and there is some vector _y_ in _E_ _n_ which is _not_ a linear combination of _u_ 1, . . . , _u_ _k_. Therefore, the vector

is nonzero. But this vector is orthogonal to all the vectors _u_ 1, . . . , _u_ _k_ , since ( _u_ _r_ , _u_ _s_ ) = _δ_ _rs_ by assumption. Since _L_ _n_ – _k_ is, therefore, of dimension ≥1, the matrix _H_ contains a unit eigenvector, _u_ _k_ +1, in _L_ _n_ – _k_.

We continue the process until we have found a full set of _n_ mutually orthogonal unit eigenvectors _u_ 1, . . . , _u_ _n_. If _U_ is the matrix with columns of these eigenvectors, then _U_ * _U_ = _I_. Moreover, _HU_ = _U_ Λ if Λ is the diagonal matrix formed from the eigenvalues λ1, . . . , λ _n_ belonging to _u_ 1, . . . , _u_ _n_. Therefore, _U_ * _HU_ = Λ, and the proof is complete.

Suppose that _H_ is a real, symmetric matrix. Does _H_ have _n_ mutually orthogonal _real_ unit eigenvectors? The answer is yes. A study of the preceding proof shows that, if _H_ is real, complex numbers need never appear, since the eigenvalues λ are real. The inner product becomes ( _x_ , _y_ ) = ∑ _x_ _i_ _y_ _i_ , without any conjugates. The unitary matrix _U_ which diagonalizes _H_ will have all components real. Thus we have:

**Theorem 3.** _If_ H _is a real_ , _symmetric matrix there is a real_ , _unitary matrix_ U _for which_ UTHU _is a diagonal matrix_ Λ _formed from the eigenvalues of_ H.

#### **PROBLEMS**

**1.** Let _H_ be the Hermitian matrix

Let _L_ 2 be the space of vectors orthogonal to the eigenvector _u_ 1 = 3−1/2 col (1, 1, 1). Define some basis for _L_ 2, and verify that _H_ maps _L_ 2 into itself. Let _B_ be defined as in the proof of Lemma 1. Find a 2 × 2 matrix _M_ such that _HB_ = _BM_. Compute an eigenvector _y_ of _M_. Hence, compute a second unit eigenvector _u_ 2 of _H_ , where ( _u_ 1, _u_ 2) = 0. Find a basis for the space, _L_ 1, of vectors orthogonal to _u_ 1 and _u_ 2. Finally, compute a third unit eigenvector, _u_ 3, orthogonal to _u_ 1 and _u_ 2. Define the unitary matrix _U_ of equation (11), and verify the identity _U_ * _HU_ = Λ.

**2.** Prove that the space _L_ _n_ – _k_ appearing in the proof of Theorem 2, has dimension _n_ – _k_ by showing that _u_ _k_ +1, . . . , _u_ _n_ provides a basis for _L_ _n_ – _k_.

**3.** If _A_ is a square _non_ -Hermitian matrix, does _A_ map the space _L_ orthogonal to an eigenvector, into itself? Illustrate.

**4.** Let _A_ be an _n_ × _n_ non-Hermitian matrix. Suppose that _A_ has distinct eigenvalues λ1, . . . , λ _n_. Show that _A_ * has the eigenvalues  . If _u_ _j_ is an eigenvector of _A_ belonging to λ _j_ and if _v_ _k_ is an eigenvector of _A_ * belonging to  , show that ( _u_ _j_ , _v_ _k_ ) = 0 if _j_ ≠ _k_ , and show that ( _u_ _j_ , _v_ _j_ ) ≠ 0. This is known as the _principle of biorthogonality_.

### **4.8 MASS-SPRING SYSTEMS; POSITIVE DEFINITENESS; SIMULTANEOUS DIAGONALIZATION**

In this section we will apply the results in the last section to mechanical systems of masses and springs, and we will generalize this application to _the simultaneous diagonalization of two quadratic forms_.

Consider the mass-spring system in Figure 4.3. We have three masses and four springs vibrating horizontally between two rigid walls. At equilibrium, the masses would have certain position coordinates, say _r_ 1, _r_ 2, and _r_ 3.

**Figure 4.3**

Let _x_ 1, _x_ 2, and _x_ 3 be the _deviations_ of the vibrating masses from their positions at rest. Thus, _x_ 1 = 0, _x_ 2 = 0, and _x_ 3 = 0 when the system is at rest. Let _z_ _i_ = _r_ i \+ _x_ i ( _i_ = 1, 2, 3). Let _l_ 01, . . . , _l_ 34 be the lengths of the four springs when they are under no tension. Then, by Hooke's law, the forces exerted by the springs are

where _k_ 01, . . . , _k_ 34 are constants, and where the two ends of the system have coordinates 0 and _l_.

Setting force equal to mass times acceleration, we find the three differential equations

where _z_ 0 = 0 and _z_ 4 = _l_. At equilibrium, the left-hand sides are zero, and _z_ _i_ = _r_ _i_. Therefore,

for _i_ = 1, 2, 3, with _r_ 0 = 0 and _r_ 4 = _l_. This is a set of three equations in the three unknown constants _r_ 1, _r_ 2, _r_ 3. We will show later that the determinant of this system is nonzero.

We wish to focus our attention on the _deviations_ from equilibrium, _x_ _i_ ( _t_ ) ( _i_ = 1, 2, 3). Subtraction of the equations (3) from the equations (2) gives

with _x_ 0 = _x_ 4 = 0. These equations may be written with matrices:

or

The real, symmetric matrices _M_ and _K_ are called, respectively, the _mass matrix_ and the _spring matrix_.

The reader should observe that a basic physical reason why the spring matrix _K_ is symmetric is that a spring is a symmetric device; it acts the same way with respect to both ends. A resistor is also a symmetric device. Therefore, a matrix _R_ = ( _r_ _ij_ ) of resistances between pairs of nodes _i_ , _j_ is a symmetric matrix. These two examples, _K_ and _R_ , illustrate why many matrices occurring in practice are symmetric.

The differential equation (6) can tell us a great deal directly, without our solving it. We shall derive a law of _conservation of energy_. The kinetic energy of our system is

To see how the kinetic energy varies, differentiate (7):

This _bilinear form_ appears if we take the inner product of the differential equation (6) with the vector _x_ ′ = _dx_ / _dt_ :

But, since _K_ is real and symmetric,

Similarly, since _M_ is real and symmetric, ( _Mx_ ′, _x_ ′)′ = 2( _Mx_ ″, _x_ ′). Therefore, (9) says

or

The term   is called _potential energy_. We shall show that this quadratic form is always positive if _x_ ≠ 0. We have

Since all _k_ _ij_ are >0, we have ( _Kx_ , _x_ ) > 0 unless 0 = _x_ 1 = _x_ 1 – _x_ 2 = _x_ 2 – _x_ 3) = _x_ 3, i.e., unless all _x_ _i_ = 0. The form   is the work which must be done to stretch our system to deviations _x_ 1, _x_ 2, _x_ 3 from equilibrium.

We shall say that a Hermitian matrix _H_ is _positive definite_ if ( _Hx_ , _x_ ) > 0 unless _x_ = 0. The examples _M_ and _K_ illustrate that positive definite matrices may arise in practice because their quadratic forms represent energy.

We will show in Theorem 2 that a change of variable _x_ = _Cy_ can simultaneously convert both energies into sums of squares:

**Theorem 1.** _A Hermitian matrix is positive definite if and only if all its eigenvalues are positive_.

_Proof_. If all the eigenvalues λ _i_ are positive, and if _U_ * _HU_ = Λ is the canonical diagonalization, then

But _y_ = 0 implies _x_ = _Uy_ = 0.

Conversely, if ( _Hx_ , _x_ ) > 0, unless _x_ = 0, then

if _u_ _i_ is a unit eigenvector belonging to λ _i_. This completes the proof.

**Theorem 2.** _Let_ M _and_ K _be_ n × n _Hermitian matrices_. _If_ M _is positive definite_ , _then there is an_ n × n _matrix_ C _for which_

_The numbers_ λ _j_ _are real_. _If_ K _is positive definite_ , _the_ λ _j_ _are positive_. _The_ λ _j_ _are generalized eigenvalues satisfying_

_If_ K _and_ M _are real_ , _then a_ real _matrix_ C, _with_ real _columns_ c _j_ , _may be found satisfying_ (17) _and_ (18).

Note that the identities (14) for the mass-spring system now follow at once if _x_ ( _t_ ) = _Cy_ ( _t_ ). In (14) all λ _i_ are positive because _K_ , as well as _M_ , is positive definite. Also _C_ and, hence, _y_ are real, since _M_ and _K_ and _x_ ( _t_ ) are real

_Proof of the Theorem_. Since _M_ is Hermitian, there is a unitary matrix _U_ such that

The effect of the transformation

is to transform the Hermitian matrix _K_ into a matrix _L_ which is also Hermitian because

Here we have used two elementary identities satisfied by the adjoint:

The proofs of these identities are as follows: For all vectors _x_ and _y_ ,

and

We now reduce the diagonal matrix (19) to the identity by a transformation which preserves the Hermitian character of _L_. Since _M_ is assumed positive definite, its eigenvalues _μ_ _i_ are positive. Taking the positive roots,  , we define the diagonal matrix

From (19) and (20), we now find

The matrix _H_ is Hermitian by the reasoning used in (21).

Without affecting _I_ , we now may reduce _H_ to a real diagonal matrix Λ by a unitary similarity transformation. If _V_ * _V_ = _I_ and _V_ * _HV_ = Λ, formula (26) yields

Setting _C_ = _UDV_ , we have the required equations (17).

The matrix _C_ has det _C_ ≠ 0, since _C_ * _MC_ = _I_ implies

Therefore, all the columns _c_ _j_ of _C_ are nonzero vectors. To obtain the eigenvalue equation (18), multiply the equation _C_ * _KC_ = Λ by ( _C_ *)−1. Since ( _C_ *)( _MC_ ) = _I_ , we have ( _C_ *)−1 = _MC_. Therefore,

Note the _generalized orthogonality_

which is implied by _C_ * _MC_ = _I_.

If _K_ , as well as _M_ , is positive definite, the numbers λ _j_ are positive because, by (29), they equal the quotients

If _K_ and _M_ are real, all the matrices _U_ , _D_ , _V_ , and Λ may be found as real matrices, according to Theorem 3 of the last section. This completes the proof of the theorem.

The simultaneous diagonalization of _M_ and _K_ provides an explicit, meaningful solution of the differential equation _Mx_ ″ + _Kx_ = 0 of mass-spring and analogous systems. Make the change of variable _x_ ( _t_ ) = _Cy_ ( _t_ ). The differential equation now becomes

Multiplication on the left by _C_ * yields the uncoupled equations

Since _K_ , as well as _M_ , is positive definite, the numbers λ _i_ are positive:  . By components, (33) reads

Therefore, we have the general solution

Since _x_ ( _t_ ) = _Cy_ ( _t_ ), where _C_ has columns _c_ _j_ ,

Equation (37) gives the solution _x_ ( _t_ ) in terms of the generalized eigenvalues λ _j_ and eigenvectors _c_ _j_. The motion _x_ ( _t_ ) is thus composed of harmonic oscillations with angular frequencies  , where λ1, . . . , λ _n_ are the roots of the polynomial equation

In fact, if Λ = diag (λ1, . . . , λ _n_ ),

The constants _α_ _j_ , _β_ _j_ may be used to fit given initial conditions

Finally, we remark that the determinant of the equilibrium equations (3) is nonzero. These equations may be written in the form _Kr_ = a given vector. But the spring matrix _K_ is positive definite, and _the determinant of a positive-definite matrix_ , being the product of its eigenvalues, _is positive_.

#### **PROBLEMS**

**1.** Let

Find the roots, λ _1_ and λ _2_ , of the characteristic equation det (λ _M_ – _K_ ) = 0.

Find nonzero vectors, _v_ 1 and _v_ 2, such that _Kv_ _i_ = λi _M_ _v_ _i_ ( _i_ = 1, 2). Find normalized vectors _c_ _i_ = _ρ_ _i_ _v_ _i_ ( _i_ = 1, 2) such that ( _Mc_ _i_ , _c_ _i_ ) = 1. Form the matrix _C_ with columns _c_ 1 and _c_ 2. Verify that _C_ * _MC_ = _I_ and _C_ * _KC_ = Λ = diag(λ1, λ2).

**2.** Let _M_ and _K_ be defined as in Problem 1. Solve initial-value problem

**3.** Let _M_ and _K_ be positive-definite Hermitian matrices. Let _x_ ( _t_ ) satisfy the differential equation

where _L_ is any matrix for which the Hermitian matrix _L_ \+ _L_ * is positive definite. Show that the total energy

decreases steadily as _t_ increases.

**4.** If a Hermitian matrix _H_ is positive definite, show that _H_ = _P_ 2, where _P_ is also positive definite.

**5.** If a Hermitian matrix _H_ is positive definite, show that it satisfies the generalized Schwarz inequality,

and the generalized triangle-inequality,

**6.** * Let _M_ = _I_ , and let

For these _three_ matrices _M_ , _K_ , and _L_ , show that _no_ two nonsingular matrices, _P_ and _Q_ , exist for which all three matrices _PMQ_ , _PKQ PLQ_ are diagonal matrices. (Hence, the differential equation _Mx_ ″ + _Lx_ ′ + _Kx_ = 0 cannot be uncoupled by setting _x_ = _Qy_ and multiplying the differential equation on the left by _P_.)

### **4.9 UNITARY TRIANGULARIZATION**

We have shown that not every square matrix is similar to a diagonal matrix. However, as we will now show, every square matrix _is_ similar to a triangular matrix. For many purposes triangularization is sufficient.

**Theorem 1.** _Let_ A _be an_ n × n _matrix_. _Then there is a unitary matrix_ U _for which_

_In other words_ T _has all zeros below the main diagonal_. _The diagonal elements_ tii _are the eigenvalues of_ A.

It is noteworthy that the similarity transformation _A_ → _T_ can always be accomplished by a _unitary_ matrix, with _U_ * = _U_ −1. If _A_ happens to be Hermitian, _T_ is necessarily diagonal because _T_ is triangular and _T_ = _U_ * _AU_ is Hermitian. Therefore, our proof will provide an independent proof that a Hermitian matrix _H_ can be diagonalized by a unitary similarity transformation _U_ * _HU_ = Λ.

_Proof of the Theorem_. The theorem is true for _n_ = 1, with 1· _a_ 11·1 = λ1 = _T_. Supposing the theorem true for _n_ ≤ _m_ , we will prove it for _n_ = _m_ \+ 1.

Let _u_ 1 be a unit eigenvector of _A_ :

The vector _u_ 1 exists because the polynomial det (λ _I_ = _A_ ) is zero for at least one complex number λ = λ1.

Since _u_ 1 ≠ 0, _u_ 1 has at least one nonzero component, say the _r_ th component. Consider the vectors

These _n_ vectors clearly form a basis for _E_ _n_. If these vectors are orthogonalized by the Gram-Schmidt process, the result is _n_ vectors

Let _V_ be the unitary matrix with columns _v_ _j_. Then _AV_ has the columns

The matrix _V_ *( _AV_ ) has the first column

which equals λ1 col (1, 0, 0, . . . , 0). Thus, _V_ * _AV_ has the form

where the numbers * are irrelevant and where _B_ is an ( _n_ − 1) × ( _n_ − 1) matrix.

By induction, we have an ( _n_ − 1) × ( _n_ − 1) unitary matrix _W_ for which _W_ * _BW_ = _T_ 1, a triangular matrix with all zeros below the main diagonal. Now form the _n_ × _n_ matrix

The matrix _Y_ is unitary because its columns are mutually orthogonal unit vectors. Multiplying (6) on the right by _Y_ and on the left by _Y_ *, we find

But _W_ * _BW_ = _T_ 1 which has all zeros below the main diagonal. Therefore, _T_ has all zeros below the main diagonal. The product of unitary matrices _U_ = _VY_ is unitary. Therefore, (8) gives the required triangularization (1).

The diagonal elements _t_ 11, . . . , _t_ _nn_ are the eigenvalues of _A_ because we have the expansion

for the determinant of the triangular matrix λ _I_ – _T_. Since the characteristic polynomial is invariant under a similarity transformation, we must also have

This theorem has many applications, of which we shall give only a few.

**Theorem 2.** _Let_ A _be an_ n × n _matrix with multiple eigenvalues. Then there is a matrix_ B _as near as we wish to_ A _such that_ B _has distinct eigenvalues_.

_Proof_. Let _A_ be given. For any _є_ > 0, we must show that there is a matrix _B_ = ( _b_ _ij_ ) with | _b_ _ij_ – _a_ _ij_ | < _є_ ( _i, j_ = 1, . . . , _n_ ) and such that _B_ has distinct eigenvalues.

From the unitary triangularization _U_ * _AU_ = _T_ , we may represent _A_ in the form

Keep _U_ fixed and the numbers * fixed, but change the eigenvalues λ1 . . . , λ _n_ into _distinct_ numbers  . The result is a new matrix

The eigenvalues of _B_ are the _distinct_ numbers   because _B_ is similar to _T_ ′. By the representations (10) and (11), if the   are chosen sufficiently near the λ _k_ we shall have | _b_ _ij_ – _a_ _ij_ | < _є_ for all _i_ , _j_. This completes the proof.

**Theorem 3.** _Let_ A _be an_ n × n _matrix. Then the powers_ Ar _tend to the zero matrix as_ r → ∞ _if and only if all the eigenvalues_ λ _k_ _of_ A _lie in the unit circle_ | λ | < 1.

This is the basic theorem of numerical matrix-iteration methods.

_Proof_. If | λ1 | ≥ 1 for any eigenvalue λ1, the powers _A_ _r_ cannot tend to zero because, if _Au_ l = λ1 _u_ 1 with _u_ l ≠ 0, then

which does not tend to the zero vector as _r_ → ∞.

Suppose, conversely, that all | λ _j_ | < 1; show _A_ _r_ → **0** as _r_ → ∞. This is trivial if _A_ has _distinct_ eigenvalues. For then, if Λ = diag (λ1, . . . , λ _n_ ),

and

which tends to zero as _r_ → ∞ because if Λ _r_ = diag  .

If _A_ has multiple eigenvalues, we still have the unitary triangularization (1). The identities

imply that _A_ _r_ → O if and only if _T_ _r_ → O. We now _majorize T_ by the triangular matrix

where _α_ ≥ all | _t_ _ij_ | for _j_ > _i_ and where _μ_ 1, _μ_ 2, . . . , _μ_ _n_ are any _n distinct_ positive numbers such that

The powers _M_ _r_ tend to O as _r_ → ∞ because _M_ has the _n distinct_ eigenvalues _μ_ _k_ inside the unit circle. But the inequalities

between _T_ and _M_ imply the inequalities

between corresponding components of the powers _T_ _r_ and _M_ _r_. Since the _i, j_ component   tends to 0 as _r_ → ∞, (19) shows that the _i_ , _j_ component of _T_ _r_ also tends to zero as _r_ → ∞. This completes the proof.

As a final application we will prove the famous Cayley-Hamilton theorem, which states that a square matrix satisfies its own characteristic equation.

**Theorem 4.** _Let A be an_ n × n _matrix. Let_

_Then_

**E XAMPLE 1.** For

we have det (λ _I_ − _A_ ) = λ2 − 5λ − 2. Now (21) becomes

or

_Proof of the Theorem_. We will later give a purely algebraic proof of this theorem. Let  (λ)= det (λ _I_ − _A_ ). We wish to prove that  ( _A_ )= 0. This is obvious if _A_ has distinct eigenvalues. For then _A_ = _C_ Λ _C_ − l, where if Λ = diag (λ1, . . . , λ _n_ ); and

But if  (Λ) = diag [ (λ1), . . . ,  (λ _n_ )] = 0, because the numbers  (λ _j_ ) are all 0.

If _A_ has multiple eigenvalues, Theorem 2 states that _A_ is the limit of matrices _B_ with distinct eigenvalues. If

then _β_ _j_ → _α_ _j_ as _B_ → _A_ because the coefficients of the characteristic polynomial are continuous, multinomial functions of the components of a matrix. Therefore,

But _B_ _n_ \+ _β_ 1 _B_ _n_ − 1 \+ · · · + _β_ _n_ _I_ = _ψ_ ( _B_ ) = 0 because _B_ has distinct eigenvalues. Therefore,  ( _A_ ) = 0.

#### **PROBLEMS**

**1.** If _A_ is an _n_ × _n_ matrix, prove that there is a unitary matrix _V_ such that _VAV_ * = _L_ , where _L_ is _lower_ triangular ( _l_ _ij_ = 0 for _j_ > _i_ ). Use Theorem 1.

**2.** Find a unitary triangularization _U_ * _AU_ = _T_ for the matrix

**3.** Find a unitary triangularization _U_ * _AU_ = _T_ for the matrix

**4.** If _U_ * _AU_ = _T_ is a unitary triangularization, show that

**5.** Consider a system of differential equations _dx_ / _dt_ = _Ax_ , where _A_ is _not_ similar to a diagonal matrix. Let _U_ * _AU_ = _T_ be a unitary triangularization of _A_. Show how the change of variable _x_ = _Uy_ allows the system to be solved recursively for _y_ _n_ , then _y_ _n_ -1, . . . , and finally _y_ 1.

### **4.10 NORMAL MATRICES**

_What is the most general class of_ n × n _matrices_ N _which have_ n _orthogonal eigenvectors_? We shall call these matrices _normal matrices_. Let λ1, . . . , λ _n_ be the eigenvalues of _N_ , and let _u_ 1, . . . , _u_ _n_ be corresponding eigenvectors satisfying ( _u_ _j_ , _u_ _k_ ) = δ _jk_. Let Λ = diag (λ1, . . . , λ _n_ ), and let _U_ be the unitary matrix with columns _u_ l, . . . , _u_ _n_. Then

Since _N_ * = _U_ Λ* _U_ *, the normal matrix _N_ is Hermitian if and only if Λ = Λ*, which means that Λ is real. Let Λ = Λ′ + _i_ Λ″ = Re Λ + _i_ Im Λ. Suppose that Λ is pure-imaginary: Λ = _i_ Λ″. Then Λ* = – Λ, and (1) shows that _N_ = – _N_ *. Suppose that all λ _k_ lie on the unit circle. Then Λ*Λ = _I_ , so that _N_ * _N_ = _I_ and _N_ is unitary. Conversely, we shall show later that every skew-Hermitian or unitary matrix is a normal matrix.

Since Λ = Λ′ + _i_ Λ″, we can write any normal matrix in the form

where _H_ ′ and _H_ ″ are commuting Hermitian matrices:

A simple property held in common by all Hermitian, skew-Hermitian, and unitary matrices is that they commute with their adjoints: _NN_ * = _N_ * _N_. We now show the converse:

**Theorem 1.** _An_ n × n _matrix_ N _is_ normal, _i.e., has a complete set of orthogonal eigenvectors, if and only if_ NN* = N*N.

_Proof_. If _N_ is normal, equation (1) gives

Supposing, instead, that _NN_ * = _N_ * _N_ , let us prove that _N_ is normal.

For any square matrix _N_ we have _N_ = _H_ ′ + _iH_ ″, where _H_ ′ and _H_ ″ are the Hermitian matrices

Since _NN_ * = _N_ * _N_ , the matrices _H_ ′ and _H_ ″ commute:

We will now show that _H_ ′ _H_ ″ = _H_ ′ _H_ ″ implies that the Hermitian matrices _H_ ′ and _H_ ″ can be diagonalized by the _same_ unitary matrix _U_. Let   be the eigenvalues of _H_ ′, and let Λ′ = diag  . Since _H_ ′ is Hermitian there is a unitary matrix _V_ for which _V_ * _H_ ′ _V_ = Λ′. Let _V_ * _H_ ″ _V_ = _K_ ″. The matrix _K_ ″ is, of course, Hermitian; but we cannot be sure that _K_ ″ is diagonal. For example, the matrices

commute; but if we take _V_ = _I_ , then _K_ ″ = _H_ ″ ≠ a diagonal matrix.

Since _H_ ″ and _H_ ″ commute, Λ″ and _K_ ″ commute:

If _K_ ″ has components _k_ _ij_ , the commuting of Λ′ and _K_ ″ implies

Therefore, _k_ _ij_ = 0 unless  .

What does this mean? If the   are distinct, then _k_ _ij_ = 0 unless _i_ = _j_ ; then _K_ ″ is diagonal. If the   are not distinct, we may write

where Λ′ consists of a string of _m_ _i_ _equal_ eigenvalues, say _γ_ _i_ , where _γ_ 1 > _γ_ 2 >... _γ_ _s_. Since _k_ _ij_ = 0 unless  , there is a corresponding partitioning

where _K_ _i_ is an _m_ _i_ × _m_ _i_ Hermitian matrix.

For each _K_ _i_ there is a unitary matrix _W_ _i_ such that   is diagonal.

Define

The matrix _W_ is an _n_ × _n_ unitary matrix. The matrix _W_ * _K_ ″ _W_ is diagonal; it consists of zeros except for the blocks   along the diagonal, and each block   is a diagonal matrix.

Form the matrix _W_ *Λ′ _W_. This matrix consists of zeros except for the blocks  . But Λ _i_ = _γ_ _i_ _I_ _i_ where _I_ _i_ is the identity matrix of order _m_ _i_.

Therefore,

Therefore, _W_ *Λ′ _W_ = Λ′. In summary, _if U_ = _VW_ , we have shown

where Λ′ and Λ″ are real diagonal matrices. From the representation _N_ = _H_ ′ + _iH_ ″ we find the required diagonalization

The columns of the unitary matrix _U_ form a complete set of orthogonal eigenvectors for _N_.

**Corollary.** _Let_ A _be an_ n × n _matrix for which_ A* = −A _or for which_ A*A = I. _Then A has_ n _mutually orthogonal unit eigenvectors_.

_Proof_. In either case, _A_ * _A_ = _AA_ *.

RIGID MOTIONS IN EUCLIDEAN SPACE

As an application of the theory of normal matrices, we shall discuss the rigid motions of real, three-dimensional space. The rigid motions _x_ → _Rx_ preserve length. We shall assume that there is no translation, so that 0 → 0. Therefore, according to Section 4.4, the rigid motions are prescribed by real, unitary matrices _R_ :

A physically realizable rigid motion can be achieved by a continuous sequence of rigid motions, _M_ ( _t_ )(0 ≤ _t_ ≤ 1), where

The eigenvalues of each matrix unitary _M_ must lie on the unit circle because _Mx_ = λ _x_ implies

Since _M_ is real, if _M_ has complex eigenvalues, they occur in complex conjugate pairs _e_ _iθ_ , _e_ – _iθ_. Therefore, the three eigenvalues of _M_ are all ±1, or they are ±1, _e_ _iθ_ , _e_ – _iθ_ , where 0 < _θ_ < _π_. In either case, det _M_ ( _t_ ) = ±1 for all _t_. But det _M_ (0) = det _I_ = 1. Since _M_ ( _t_ ) varies continuously, det _M_ ( _t_ ) varies continuously. Therefore, det _M_ ( _t_ ) = 1 for all _t_ , since a continuous function cannot jump from +1 to – 1.

In particular,

Therefore, the eigenvalues of _R_ have one of these forms:

Let _U_ be a unitary matrix which diagonalizes the normal matrix _R:_

_Case_ ( _α_ ). In this case Λ = _I_ , and

_Case_ ( _β_ ). Since the normal matrix _R_ has real eigenvalues, _R_ is Hermitian.

For example,

has the eigenvalues 1, – 1, −1. Then the real, Hermitian matrix _R_ can be diagonalized by a _real_ unitary matrix _U_. This was proved in Theorem 3 of Section 4.7. Let the columns of _U_ be the real unit vectors _u_ , _v_ , and _w_. Since

we see that _R_ is a rotation about the _u_ axis, through 180°, in the _v_ , _w_ plane.

_Case_ ( _γ_ ). Since there is a single eigenvalue λ = 1, there is a real eigenvector _u_ for which _Ru_ = _u_. Every other eigenvector belonging to λ = 1 is a scalar multiple of _u_. Let the complex unitary matrix _U_ have columns _u_ , _v_ , and _w_. Since

while taking conjugates gives

the vector _w_ is a scalar multiple of   we shall take _w_ =  . Let the vector _v_ have real part _a_ and imaginary part _b_ : _v_ = _a_ \+ _ib_. Then

implies

Since _v_ and _w_ are orthogonal, we have

But ( _a_ , _b_ ) = ( _b_ , _a_ ) since the vectors _a_ and _b_ are real. Therefore,

But 1 = ( _v_ , _v_ ) implies

Therefore _a_ and _b_ are real, orthogonal unit vectors with  . Since _u_ is orthogonal to _v_ = _a_ \+ _ib_ , the real vectors _u_ , _a_ , and _b_ are mutually orthogonal. Let _V_ be the _real_ unitary matrix with columns _u_ ,  , and  . Then, by (12),

Thus, _R_ is a rotation about the _u_ axis, through the angle _θ_ , in the _a_ , _b_ plane.

In summary, a real 3 × 3 unitary matrix, _R_ , with det _R_ = 1, represents a _rotation_. The eigenvalues and eigenvectors give the angle of rotation, the axis of rotation, and two orthogonal vectors in the plane of rotation.

#### **PROBLEMS**

**1.** Find the axis of rotation in the rigid motion given by the matrix

**2.** Find the angle of rotation, the axis of rotation, and two orthogonal vectors in the plane of rotation for the rigid motion

**3.** * Consider the differential equation _dx_ ( _t_ )/ _dt_ = _Nx_ ( _t_ ). This equation has the solution _x_ ( _t_ )= _X_ ( _t_ ) _x_ (0), where _X_ ( _t_ ) is the exponential matrix _e_ _Nt_ (See Section 3.4). If _N_ is a normal matrix, show that _X_ ( _t_ ) is normal for each time, _t_. If _N_ is Hermitian, show that _X_ ( _t_ ) is Hermitian. If _N_ is skew-Hermitian, show that _X_ ( _t_ ) is unitary.

**4.** Let _N_ * _N_ = _NN_ *. Show that there are a positive-semidefinite Hermitian matrix, _P_ , and a unitary matrix, _V_ , such that _N_ = _PV_ = _VP_. (Use the factorization _N_ = _U_ Λ _U_ *. If _NN_ * = _U_ ΛΛ* _U_ * = _P_ 2, how can _P_ be defined?)

**5.** Let _T_ be a triangular matrix, with _t_ _ij_ = 0 for _i_ > _j_. Show that _TT_ * = _T_ * _T_ only if _T_ is a diagonal matrix. Hence, give an independent proof of Theorem 1 based on the unitary triangularization _N_ = _UTU_ *.

## **5 THE JORDAN CANONICAL FORM**

### **5.1 INTRODUCTION**

Not every matrix can be diagonalized by a similarity transformation. This fact has an important consequence in the theory of differential equations : Not every system of differential equations

has a solution which is a linear combination of exponential functions

For example, the problem

has the nonexponential solution

The matrix _A_ in the example (3) cannot be diagonalized by a similarity transformation

If (5) were possible, then we must have λ1 = λ2 = 7 because _A_ and Λ must have the same characteristic polynomial. Then (5) implies the false equation

To obtain the solutions of differential equations (1) in all cases, we may use the following theorem of C. Jordan:

**Theorem 1.** _Let_ A _be an_ n × n _matrix whose different eigenvalues are_ λ1, . . . , λs with multiplicities _m_ 1, . . . , _m_ s:

_Then_ A _is similar to a matrix of the form_

where Λ _i_ _is an_ m _i_ × m _i_ _matrix of the form_

_with each_ * _equal to_ 0 _or_ 1. (Some *'s may equal 0 while other *'s equal 1.)

This chapter will be devoted to the proof of Jordan's Theorem, which is a generalization of the preceding results. If _A_ has _distinct_ eigenvalues, we have _s_ = _n_ and all _m_ _j_ = 1 in (6). Then the matrix Λ _i_ is the l × l matrix λ _i_ , and _J_ = diag (λ1, . . . , λ _n_ ). If _A_ has multiple eigenvalues _but_ has _n_ linearly independent eigenvectors (e.g., if _A_ is Hermitian), then _A_ is similar to _J_ with every number * equal to 0.

Let us show how Jordan's Theorem applies to differential equations (1). If _C_ −1 _AC_ = _J_ , we make the change of variable _x_ ( _t_ ) = _Cy_ ( _t_ ). Then (1) becomes

We now partition the _n_ -component column vector _y_ ( _t_ ) into parts

Similarly, we partition the initial vector   into _s_ parts:  , , . . . , . The block form (7) for _J_ now yields _s disjoint_ systems

For _i_ = 1, equation (11) involves only the first _m_ 1 components of _y_ ; for _i_ = 2, (11) involves only the next _m_ 2 components of _y_ , etc. Therefore, the _s_ systems (11) may be solved separately.

Every one of the systems (11) has the form

where

with * = 0 for _k_ = 1. Make the change of variable

Then (12) becomes

Since * = 0 for _k_ = 1, (15) yields _w_ 1( _t_ ) = constant. Integrating (15) successively gives

_The Jordan canonical form thus shows that, if A has the different eigenvalues_ λ1, . . . , λ _s_ _with muliplicities_ m1, . . . , ms, _then the vector_ x(t) _solving_ x′(t) = Ax(t), x(0) = b, _has components of the form_

**E XAMPLE 1.** Let _A_ be similar to the matrix

Here _n_ = 4, λ1 = 2, λ2 = 3, _m_ 1 = 2, _m_ 2 = 2,

The differential equation _dy_ / _dt_ = _Jy_ can be solved by setting

The equations (11) are the independent systems

If _y_ 1 (0) = 1, . . . , _y_ 4(0) = 4, we find

Suppose that _x_ ( _t_ ) is related to _y_ ( _t_ ) by the equation

Then, for example, for _x_ 3 ( _t_ ) we find

where

#### **PROBLEMS**

**1.** Suppose that _y_ ( _t_ ) satisfies the differential equation

Solve for _y_ ( _t_ ). In this example identify the matrices Λ _i_ and the vectors _y_ ( _i_ )( _t_ ).

**2.** Let _y_ ( _t_ ) be the solution of Problem 1. Suppose that _x_ ( _t_ ) = _Cy_ ( _t_ ), where

In this example, identify all the different numbers _s_ , _m_ _i_ , _ξ_ _vij_ , λ _i_ , _n_ in the general formula (17).

**3.** Assume that Jordan's Theorem is true. Let _p_ be some very large positive number, and let _D_ = diag (1, _p_ , _p_ 2, . . . , _p_ _n_ −1). If we form the matrix _D_ –l _JD_ , describe the resulting modified Jordan form for the matrix _A_. What happens when _p_ → ∞?

### **5.2 PRINCIPAL VECTORS**

As in the preceding section, assume that _A_ is an _n_ × _n_ matrix with the different eigenvalues λ1, . . . , λ _s_ , with multiplicities _m_ 1, . . . , _m_ _s_. In this section we will define _principal vectors_ as generalizations of eigenvectors. We say that a zero or nonzero vector _p_ is a _principal vector of grade_   0 _belonging to the eigenvalue_ λ1 if

and if there is no smaller non-negative integer   for which (λ _i_ _I_ – _A_ )γ _p_ = 0.

The vector _p_ = 0 is the principal vector of grade 0. The eigenvectors are the principal vectors of grade 1.

We shall define the linear space   as the linear space of all principal vectors of grade   belonging to λ _i_. Thus,

and   is the null space of the matrix  . We have

because

If _A_ has distinct eigenvalues, i.e., if _s_ = _n_ and all _m_ 1 = 1, we know from Section 4.1 that _A_ has _n_ linearly independent eigenvectors _c_ 1, . . . , _c_ _n_. Then any _x_ has an expansion

where _p_ _i_ = _ξ_ _i_ _c_ _i_ is a principal vector belonging to λ _i_ ; the grade of _p_ _i_ is 1 if _ξ_ _i_ ≠ 0, or 0 if _ξ_ _i_ = 0. In either case, _p_ _t_ lies in the space _P_ 1(λ _i_ ); recall that 1 = _m_ _i_.

In the general case, in which _m_ _i_ may be > 1, we have a _representation by principal vectors:_

**Theorem 1.** _Let_ A _be an_ n × n _matrix with the different eigenvalues_ λ1, . . . , λs _with multiplicities_ m1, . . . , ms. _Then every n-component column vector_ x _has a representation_

_where p_ 1 _is a uniquely defined principal vector belonging to_ λi _of grade_   mi.

We shall prove this theorem after some algebraic preliminaries.

**E XAMPLE 1.** Let

There are two eigenvalues, λ1 = 1 and λ2 = 2. We have _m_ 1 = 2, _m_ 2 = 1. The eigenvectors belonging to λ1 = 1 satisfy

Thus, the eigenvectors of λ1 are the nonzero multiples of col (0, 1,0). The eigenvectors belonging to λ2 = 2 are the nonzero multiples of col (0, 0, 1). Thus, a representation (4) of _x_ by eigenvectors is impossible if _x_ 1 ≠ 0. Since _m_ 1 = 2, we look for principal vectors of grade   2 belonging to λ1. They satisfy the equation

Thus we have

while for the second eigenvalue λ2 = 2 of multiplicity _m_ 2 = 1,

Theorem 1 merely states that every _x_ = col ( _α, β, γ_ ) has a unique representation _x_ = _p_ (1) \+ _p_ (2).

**Lemma 1.** _If _ (λ) = det (λ _I_ – _A_ ), _then _ ( _A_ ) = O.

_Proof_. This is the Cayley-Hamilton Theorem, which was stated, illustrated, and proved in Section 4.9. Here, for the sake of completeness, we shall give an independent, purely algebraic proof. Define the _n_ × _n_ matrix of signed cofactors

For any square matrix, _M_ , we have _M_ (cof _M_ ) _T_ = (det _M_ ) _I_. Therefore,

If _A_ is an _n_ × _n_ matrix, every component of _C_ (λ) is a polynomial of degree   _n_ − 1 in λ. Then we may write

where each _C_ _i_ is an _n_ × _n_ matrix of constants. If

then identification of the coefficients of equal powers of λ in (10) gives

Multiply the first equation by _A_ _n_ on the left, the second by _A_ _n_ −1, etc., and add. The result is

All terms on the left-hand side of (14) cancel, giving 0 =  ( _A_ ) _I_.

Lemma 2. _Let_ s   2. _Let_  1(λ), . . . ,  s (λ) _be polynomials. Suppose thai there is no number_ λ0 _which is a root of all these polynomials. Then there are polynomials ψ_ 1(λ), . . . , ψs(λ) _for which_

Note that the identity (15) is clearly impossible if for some λ0,

**E XAMPLE 2.** The polynomials

have no common root. One may verify the identity

_Proof of Lemma 2_. Let

If _N_ = 0, all  _i_ (λ) are constants, and at least one of these constants, say  _k_ is nonzero. Then the identity (15) holds if we let _ψ_ _k_ (λ) be the constant l/  _k_ and let all other ψ _i_ (λ) ≡ 0.

If _N_   1, assume without loss of generality that

Let  _k_ (λ) be the first one of the polynomials  1 2, . . . ,  _s_ which is not identically zero. Then _k_ < _s;_ otherwise, if _k_ = _s_ , every root of  _s_ would be a root of all  1. If  _k_ (λ) = constant, we set ψ _k_ (λ) ≡ 1/  _k_ and set all other _ψ_ _i_ (λ) ≡ 0. If  _k_ (λ) is not a constant, we divide all other  _i_ (λ) by  _k_ (λ) to obtain identities

If _i_ < _k_ , then  _i_ (λ) ≡ _r_ _i_ (λ) ≡ 0. If _i_ > _k_ , the degree of the remainder _r_ _i_ (λ) is less than the degree of  _k_ (λ), and therefore less than the degree of  _i_ (λ).

Now consider the polynomials

These polynomials cannot have a common root λ0; otherwise λ0 would, by (18), be a root of all the polynomials  1, . . . ,  _s_. Further, the sum of the degrees of the polynomials (19) is less than the sum of the degrees of  _i_ , . . . ,  _s_. By induction on the sum of the degrees, we may assume that there are polynomials   such that

By (18), this identity becomes

which gives the required identity (15) if we define

and

_Proof ofTheorem 1_. If _s_ = 1 and _m_ l = _n_ , then det (λ _I_ – _A_ ) = (λ – λ1) _n_ , and the Cayley-Hamilton Theorem states that

Then every _x_ is, itself, a principal vector of grade   _m_ 1 = _n_ belonging to λ1 and (5) holds with _s_ = 1 and _p_ (l) = _x_.

If _s_   2, define the polynomials

These polynomials have no common root. By Lemma 2, there are polynomials _ψ_ _i_ (λ) for which

If ω(λ) = a polynomial ≡ 1, we must have ω( _A_ ) = _I_ for any square matrix _A_ , since the permissible manipulations of addition, subtraction, and multiplication are the same for a scalar variable λ as for a single square matrix _A_. For example,

implies

Thud,(25) implies

Multiplying the matrix (26) into the vector _x_ gives

But each [ ] is a principal vector! In fact,

But [ ] =  ( _A_ )= 0 by the Cayley-Hamilton Theorem, where

Thus, the vector (28) is zero, and

is a principal vector of grade   _m_ i belonging to λ _i_. Thus, (27) is the required representation (5).

To show that the representation (5) is unique, suppose

where _q_ 1 ≠ _p_ 1, and where each _q_ _i_ is a principal vector of grade   _m_ i belonging to λ _i_. Taking (5) – (5′), we find

where _r_ _i_ = _p_ _i_ − _q_ _i_. Now _r_ 1 is a nonzero principal vector belonging to λ1. Let _r_ 1 have grade  . Let

Then _c_ 1 is an _eigenvector_ belonging to λ1 since ( _A_ − λ1 _I_ ) _c_ l = 0. Multiply (31) by the matrix

The result is

This is a contradiction because the right-hand side is nonzero. Note the use of the identity  1( _A_ ) _c_ 1 =  1(λ1) _c_ 1 for the eigenvector _c_ 1 in the derivation of (33).

#### **PROBLEMS**

**1.** Verify the Cayley- Hamilton Theorem for the matrix

**2.** Suppose that det (λ _I_ – _A_ ) =  (λ) = (λ – 1)4(λ – 2)(λ2 \+ l)3. Identify the polynomials  _i_ (λ) discussed in the proof of Theorem 1.

**3.** Identify the spaces   for the matrix

Also identify the principal vectors of grade 2.

**4.** For the matrix _A_ in Problem 3, find the unique representations of the vectors

as sums of principal vectors: _x_ = _p_ (1) \+... + _p_ ( _s_ ).

**5.** Let _A_ be defined as in Problem 3, and let _A_ 1 = _TAT_ −1, where

Identify the spaces   for the matrix _A_ 1. Also identify the principal vectors of grade 2.

**6.** In general, what relationship exists between the principal vectors of a matrix _A_ and the principal vectors of a similar matrix _A_ 1 = _TAT_ –l?

**7.** * Explain how the method of proof of Lemma 2 can be made into a constructive procedure. Illustrate by finding polynomials _ψ_ 1, _ψ_ 2, _ψ_ 3 such that ∑ _ψ_ 1(λ) 1(λ) ≡ 1, where

**8.** Prove that a matrix cannot have principal vectors of grade greater than the greatest of the multiplicites of its eigenvalues.

### **5.3 PROOF OF JORDAN'S THEOREM**

The notation in this section will be that of the preceding two sections; _A_ is an _n_ × _n_ matrix with the different eigenvalues λl, . . . , λ _s_ with multiplicities _m_ 1, . . . , _m_ _s_.

**Lemma 1.** _For_ i = 1, . . . , s _let_ **B 1** _be any basis for the linear space of principal vectors of grade_   _belonging to_ λ1. _Then the collection of vectors_ B1, B2, . . . , B _s_ _is a basis for the_ n- _dimensional space_ En.

_Proof._ According to Theorem 1 of the last section each _x_ in _E n_ has a _unique_ representation

where _p_ _i_ is a principal vector of grade   belonging to λ _i_. Since _B_ _i_ is a basis, _p_ _i_ has a unique representation as a linear combination of the vectors comprising _B_ _i_. Now (1) implies that each _x_ in _E_ _n_ has a unique representation of the vectors comprising _B_ 1, _B_ 2, . . . , and _B_ _s_. Note that the vectors in _B_ 1, . . . , _B_ _s_ are linearly independent because _x_ = 0 has the unique representation (1) with all _p_ _i_ = 0.

This result allows us to consider _separately_ the spaces of principal vectors belonging to λ1, . . . , λ _s_. For any λ _i_ define the matrix

and define the linear spaces

where _m_ = _m_ _i_ = the multiplicity of λ _i_. In the notation of the lemma,

_B_ _i_ is a basis for the linear space _P_ _m_ of principal vectors of grade   belonging to the eigenvalue λ _i_ of _A_.

We shall call the basis

a _Jordan basis_ for   if the vectors _v_ _j_ can be arranged in chains as follows:

where

and where each chain has length  . The vector at the bottom of each chain is an eigenvector of _M_ belonging to the eigenvalue 0. A vector next to the bottom is a principal vector of grade 2, etc.

**E XAMPLE 1.** Let

_M_ has the eigenvalue 0 with multiplicity _m_ = 5 and has another eigenvalue of 7. We assert that

is a Jordan basis _J_ for the space _P_ 5 of vectors _x_ satisfying _M_ 5 _x_ = 0. In fact, (5) takes the form

where

In this example, _P_ 5 is the space of vectors in _E_ 6 whose last component equals 0. In fact, the matrix _M_ 5 consists of zeros except for the number 75in the lower-right corner.

**Lemma 2.** _The space_ P _m_ _defined in_ (3) _has a Jordan basis_.

_Proof_. If _m_ = 1, _P_ 1 is the set of vectors _x_ satisfying _Mx_ = 0. If _x_ 1, . . . , _x_ _z_ is a basis for _P_ l each _x_ _j_ is an eigenvector of _M_ ; therefore, any basis for _P_ 1 is a Jordan basis.

If _m_ > 1, we make the following assertion: _Let_ y1, . . . , y _β_ _be any basis for_ Pg-i, _where_  . _Let a basis_

_for_ Pg _be formed by appending any necessary additional vectors_ x1, . . . , x _α_ , _where_  . (If _α_ = 0,   and there are no _x_ 's.) _Then the vectors_ y1, . . . , y _β_ _may be replaced in_ (7) _by vectors_ z1, . . . , z _β_ _such that_

_is a_ Jordan basis for  . The assertion, which we will prove by induction, yields the lemma when  .

Let us prove the assertion when  . If _α_ = 0, we have _P_ 2 = _P_ 1, and the required Jordan basis (8) is found simply by setting _z_ _i_ = _y_ _i_ ( _i_ = 1, . . . , _β_ ). Suppose _α_ > 1. Form the vectors _Mx_ 1, . . . , _Mx_ _α_. We assert that the 2 _α_ vectors _x_ 1, . . . , _x_ _α_ , _Mx_ 1, . . . , _Mx_ _α_ are independent. Suppose

Multiplication by _M_ gives

since _M_ 2 _x_ _i_ = 0, _x_ _i_ being in _P_ 2. Now (10) states that _a_ 1 _x_ 1 \+ . . . + _a_ _α_ _x_ _α_ lies in _P_ 1. Therefore, ∑ _a_ _i_ _x_ _i_ equals some linear combination of _y_ 1, . . . , _y_ _β_ (a basis for _P_ 1). Therefore, all _a_ _i_ = 0, since _x_ l, . . . , _x_ _α_ , _y_ 1, . . . , _y_ _β_ are given as independent.

Now (9) states _M_ (∑ _b_ _i_ _x_ _i_ ) = 0. Then ∑ _b_ _i_ _x_ _i_ is some combination of _y_ 1, . . . , _y_ _β_. Since _x_ 1 , . . . , _x_ _α_ _y_ 1, . . . , _y_ _β_ are independent, all _b_ _i_ = 0 in (9). This proves the independence of the 2 _α_ vectors _x_ _i_ , _Mx_ _i_.

The vectors _Mx_ 1, . . . , _Mx_ _α_ are independent vectors in _P_ 1. If they do not span _P_ 1 adjoin vectors _z_ _α_ +1, . . . , _z_ _β_ so that _Mx_ 1, . . . , _Mx_ _α_ , _z_ _α_ +1, . . . , _z_ _β_ are a basis for _P_ 1. Now the required Jordan basis for _P_ 2 is

In the notation of (8), _z_ 1 = _Mx_ 1, . . . , _z_ _a_ = _Mx_ _α_.

Finally, we prove the assertion for  . If _α_ = 0,   and the assertion follows by induction, since there is a Jordan basis for  . Suppose  . Form the vectors _Mx_ 1, . . . , _Mx_ _α_. Again we assert that the 2 _α_ vectors _x_ 1, . . . , _x_ _α_ , _Mx_ 1, . . . , _Mx_ _α_ are independent. For suppose ∑ _a_ _i_ _x_ _i_ \+ ∑ _b_ _i_ _Mx_ _i_ = 0. Multiplication by   gives

since all  . Now (12) implies that ∑ _a_ _i_ _x_ _i_ is a combination of the _y_ 's. Since the _α_ \+ _β_ vectors ( _x_ 1, _y_ _j_ ) are independent, all _a_ _i_ = 0.

We now have just ∑ _b_ _i_ _Mx_ _i_ = 0. Therefore, ∑ _b_ _i_ _x_ _i_ lies in  . Therefore, ∑ _b_ _i_ _x_ _i_ is a combination of the _y_ 's. Therefore, all _b_ i = 0. This proves the independence of the 2 _α_ vectors _x_ 1, . . . , _x_ _α_ , _Mx_ 1, . . . , _Mx_ _α_.

Let _w_ 1, . . . , _w_ _γ_ be any basis for  . The vectors

lie in  . Further, these vectors are independent because

implies  , which implies all _a_ _i_ = 0, as before; now (14) states ∑ _b_ _i_ _w_ _i_ = 0, which implies all _b_ _i_ = 0.

If the independent vectors (13) are not a basis for  , adjoin certain vectors _q_ 1, . . . , _q_ δ so that the combined list

is a basis for  . By induction, we can replace _w_ 1, . . . , _w_ _γ_ in (15) by a new basis _z_ 1, . . . , _z_ _γ_ for   so that the new list

is a _Jordan_ basis for  .

We now replace the original basis _y_ 1, . . . , _y_ _β_ of   by the Jordan basis (16) for  . Then

is a Jordan basis for  . This completes the proof of the lemma.

**E XAMPLE 2.** Let

The vectors

are a basis for the space _P_ 1. Consider the basis for _P_ 2:

To construct a Jordan basis, form

Define _z_ 2 in any way so that _z_ 1, _z_ 2 are a new basis for _P_ 1; say _z_ 2 = col (0, 0,1). Then

is a Jordan basis for _P_ 2.

**Lemma 3.** Let λ _i_ _be an eigenvalue of multiplicity_ mi _belonging to the_ n × n _matrix_ A. Let Bi _be a_ Jordan _basis for the set of vectors_ × _such that_

_then, if we regard_ Bi _as the matrix whose columns are the basis vectors_ ,

_where_ Λi _has the form_

_proof_. Let _M_ = A – λ _i_ _I_ , and let the Jordan basis _B_ _i_ be given by the chains (5). Thus

By (5) we have

But _A_ = λ _i_ _I_ \+ _M_. Therefore,

But this matrix equals

proveded that the sequence of *'s in (20) is chosen as follows:

Observe that there are _z_ – 1 numbers *. The proof is complete, since (23) has the required form _B_ _i_ Λ _i_. Please observe that we have not yet proved that the number of colmns in _B_ _i_ equals _m_ _i_.

**E XAMPLE 3.** Suppose that

is a Jordan basis of the space of all vectors _x_ satisfying

Then _B_ _i_ has _n_ rows and seven columns:

We have

But this equals _B_ _i_ Λ _i_ if we define the 7 × 7 matrix

_Proof of Jordan's Theorem._ Now we can prove Theorem 1 of Section 5.1. For _i_ = 1, . . . , _s_ let _B_ _i_ be the matrix whose columns are a _Jordan_ basis for the principal vectors of grade   belonging to the eigenvalue λ _i_ of _A_. We have just derived the identity _AB_ _i_ = _B_ _i_ Λ _i_ where Λ _i_ is a square matrix of the form (20).

Form the matrix

By Lemma 1, the columns of this matrix are a basis for _E_ _n_. The matrix _B_ is, therefore, an _n_ × _n_ matrix with an inverse _B_ −l. Multiplication by _A_ gives

But this equals

Multiplication of the identity _AB_ = _BJ_ by _B_ −l gives the required result, _B_ −1 _AB_ = _J_.

It remains only to show that Λ _i_ is an _m_ _i_ × _m_ _i_ matrix. That follows immediately from the just-proved similarity of _A_ and _J._ We have

or

by the block-diagonal form of _J_. Let Λ _i_ be an _o_ _i_ × _o_ _i_ matrix. Then the triangular matrix Λ _i_ has characteristic determinant

Since λ1, . . . , λ _s_ are the different eigenvalues of the _n_ × _n_ matrix _A_ , the identity (28) implies _m_ _i_ = _o_ _i_.

We conclude further that _the multiplicity_ , m _i_ , _of_ λ _i_ _equals the dimension of the space of principal vectors of grade_   _belonging to_ λ _i_. This follows because the number of columns in _B_ _i_ equals the order of Λ _i_ in the identity (19).

#### **PROBLEMS**

**1.** Consider the matrix

What are the eigenvalues? What are the eigenvectors? What are the principal vectors? Is _v_ 1 = col [1,0] a principal vector of grade 2? Form _v_ 2 = _Mv_ 1 = ( _A_ – λ1 _I_ ) _v_ 1. Let _B_ be the matrix whose columns are _v_ l and _v_ 2. Compute _J_ = _B_ −1 _AB_.

**2.** Let

Find Jordan bases _B_ 1 and _B_ 2 corresponding to the two different eigenvalues of _A_. Let _B_ =  _B_ 1, _B_ 2], as in [(25). Compute _J_ = _B_ −l _AB_.

**3.** Let

Show that _J_ 1 and _J_ 2 cannot be Jordan canonical forms for the same matrix _A_. (Discuss the dimension of the space _P_ 1.)

**4.** Let

and let _J_ 1 be defined as in the last problem. Show that _J_ 1 and _J_ 3 are Jordan forms of a single matrix, _A_.

**5.** * To what extent is the Jordan form of a given matrix, _A_ , uniquely defined?

## **6 VARIATIONAL PRINCIPLES AND PERTURBATION THEORY**

### **6.1 INTRODUCTION**

In this chapter we shall be concerned with maximum principles, with minimax principles, and with other variational properties of eigenvalues and eigenvectors. These principles have mathematical interest, and they often have physical significance.

Perturbation theory is concerned with what happens to the eigenvalues and eigenvectors of a matrix when the elements of the matrix are varied slightly. This topic is important in numerical computation. Seldom in science or engineering are we given a matrix with perfect accuracy. From a given matrix of data we compute eigenvalues and eigenvectors. We must be able to estimate the error in the computed eigenvalues and eigenvectors which results from error in the data.

### **6.2 THE RAYLEIGH PRINCIPLE**

Let _H_ be a Hermitian matrix. We have shown that _H_ has real eigenvalues   and has mutually orthogonal unit eigenvectors _u_ 1, _u_ 2, . . . , _u_ _n_ :

The eigenvalues of _H_ are thus values of the quadratic form ( _Hu_ , _u_ ) defined on the unit sphere || _u_ || = ( _u_ , _u_ )½ = 1:

Thus, λ1 is the greatest of these _n_ values of the quadratic form on the unit sphere. It is natural to ask whether _λ_ 1 is the greatest of _all_ values of the quadratic form ( _Hu_ , _u_ ) on the unit sphere || _u_ || = 1.

**Theorem 1.** _Let_ H _be Hermitian_. _Let_ λ1 _be the largest eigenvalue of_ H.

_Then_

_and the maximum is achieved when_ u _is an eigenvector belonging to_ λ1.

_Proof_. Let _u_ 1, . . . , _u_ _n_ be a complete set of eigenvectors satisfying (1). These vectors form a basis for _E_ _n_. Then every unit vector _u_ has a representation

where

But _Hu_ = Σ _c_ _i_ λ _i_ _u_ _i_. Therefore,

with equality in (6) when _c_ 2 = _c_ 3 = . . . = _c_ _n_ = 0 and _c_ 1 = 1. Thus   with equality when _u_ = 1. _u_ l \+ 0 + . . . + 0 in (4).

The proof is complete. We observe that if _H_ is _real_ and symmetric, the maximum (3) may be taken over _real_ unit vectors _u_ because _H_ has a complete set of real eigenvectors _u_ _i_.

**Corollary:** _If_ H _is Hermitian and_ λ1 _is its greatest eigenvalue_ , _then_

_Proof_. This follows from the homogeneity of the form. If _x_ ≠ 0 then, if _ρ_ = || _x_ || > 0,

where _u_ is the unit vector _ρ_ −1 _x_. Thus, every _Rayleigh quotient_ ( _Hx_ , _x_ )/( _x_ , _x_ ) equals some value ( _Hu_ , _u_ ) of the quadratic form at a point _u_ on the unit sphere || _u_ || = 1. Conversely, if || _u_ || = 1 then ( _Hu_ , _u_ ) = ( _Hu_ , _u_ )/( _u_ , _u_ ). The maxima (7) and (3) are, therefore, identical.

**E XAMPLE 1.** Let

Observe that _H_ = _H_ *. The eigenvalues satisfy the equation det (λ _I_ − _H_ ) = λ2 − 6λ, − 1 = 0. Therefore

According to Rayleigh's principle, for all complex _x_ 1 and _x_ 2 except 0, 0,

Equality is attained when _x_ 1 and _x_ 2 are the components of an eigenvector belonging to the eigenvalue λ1.

**E XAMPLE 2.** Suppose that A is _not_ Hermitian but has real eigenvalues   Does Rayleigh's principle still hold? Let

Then λ1 = 2 and λ1 = 1. But

if _x_ 1 = _x_ 2 = 1. _Rayleigh's principle applies only to Hermitian matrices_.

Having determined the maximum eigenvalue λ1 of a Hermitian matrix as the maximum of ( _Hu_ , _u_ ) for || _u_ || = 1, how can we find the lower eigenvalues  ? Each of these numbers is a value of ( _Hu_ , _u_ ) for || _u_ || = 1 _and u_ perpendicular to the eigenvectors belonging to the larger eigenvalues. Thus, for  , λi = ( _Hu_ , _u_ ) when _u_ = _u_ _i_ , a unit vector orthogonal to _u_ 1, . . . , _u_ _i_ −1.

**Theorem 2.** _Let_   _be the eigenvalues of a Hermitian matrix_ H. _For_  , _let_ u1, . . . , ui-1 _be mutually orthogonal unit eigenvectors belonging to_ λ1, . . . , λi−1. _Then_

_and the maximum is achieved when_ u _is an eigenvector belonging to_ λ _i_.

_Proof_. Let _u_ 1, . . . , _u_ _n_ be a complete set of eigenvectors:

For any _u_ in _E_ _n_ we may use the basis _u_ 1, . . . , _u_ _n_ for a representation

We then have

The constraints in the maximum problem (12) are

Then

But ( _u_ _j_ , _u_ _k_ ) = _δ_ _jk_. Therefore (17) becomes

Equality holds if _c_ _i_ = 1 and all other _c_ _j_ = 0, i.e., if _u_ = _u_ _i_. If λ _i_ = . . . =  , equality in (18) holds _only_ if

Even if _r_ > _i_ , an optimal vector

is an eigenvector belonging to the multiple eigenvalue λ _i_ , and _u_ is orthogonal to _u_ 1, . . . , _u_ _i_ −1.

#### **PROBLEMS**

**1.** If λ _n_ is the least eigenvalue of a Hermitian matrix, _H_ , show that

**2.** Let a Hermitian matrix _H_ have eigenvalues  , with associated unit orthogonal eigenvectors _u_ 1, . . . , _u_ _n_. For   show that λ _k_ = min ( _Hu_ , _u_ ) for all unit vectors _u_ which are orthogonal to _u_ _k_ +1, . . . , _u_ _n_ ; and show that the minimum is achieved for _u_ = _u_ _k_.

**3.** Let _N_ be any _n_ × _n_ matrix with _n_ unit orthogonal eigenvectors _u_ 1, . . . , _u_ _n_. Show that, for || _u_ || = 1, the complex variable ( _Nu_ , _u_ ) achieves its maximum absolute value when _u_ is an eigenvector of _N_ belonging to an eigenvalue of maximum absolute value.

**4.** Let _K_ and _M_ be Hermitian matrices, and let _M_ be positive definite. Let the vectors _c_ 1, . . . , _c_ _n_ and the real numbers   satisfy

(Refer to Section 4.8, Theorem 2). Prove that

Show that the maximum is achieved when _x_ = _c_ 1.

**5.** In the notation of Problem 4, prove that for _k_ > 1

for all vectors _x_ such that

Show that the maximum is achieved when _x_ = _c_ _k_.

**6.** Let _A_ be a _non_ -Hermitian matrix with eigenvalues λ _j_. Show that all λ _j_ satisfy

**7.** Let _H_ be the matrix

If _H_ has the eigenvalues  , find a numerical lower bound for λ1 by Rayleigh's principle. Find a numerical upper bound for λ3. Use several trial vectors.

### **6.3 THE COURANT MINIMAX THEOREM**

The Rayleigh principle for the second eigenvalue, λ2, of a Hermitian matrix, _H_ , states that

where _u_ l is an eigenvector belonging to the greatest eigenvalue, λ1 Thus, the Rayleigh principle does not give an independent characterization of λ2; the maximum problem for λ2 is stated in terms of an unknown eigenvector belonging to λ1.

Courant found a characterization of λ2 which does not explicitly depend upon _u_ 1. For an arbitrary vector _v_ l we may define a quantity

This maximum is a _function_ of the given vector _v_ 1. As _v_ 1 ranges over _E_ _n_ , the function _ϕ_ ( _v_ 1) takes various real values; Courant's principle states that λ2 is the _least_ of these values:

In other words, ϕ( _v_ l) is minimized when _v_ 1 = _u_ 1.

**E XAMPLE 1.** Let

Since _H_ = diag (, , ) is a _real_ Hermitian matrix, we may restrict _u_ to the class of _real_ vectors. The matrix _H_ has eigenvalues λ1 = 3, λ2 = 2, λ3 = 1. Corresponding eigenvectors are _u_ l = _e_ 1 = col (1, 0, 0), _u_ 2 = _e_ 2, _u_ 3= _e_ 3. Rayleigh's principle for λ2 is

But ( _u_ , _e_ l) = _u_ 1. Therefore,

To use Courant's principle, we must minimize

We have _ϕ_ (0) = λ1 = 3. If _v_ l ≠ 0, then _ϕ_ ( _v_ 1) is the largest value of ( _Hu, u_ ) on the unit circle which is the intersection of the unit sphere with the plane through the origin perpendicular to _v_ 1. To minimize the effect of the large coefficient 3 of   in ( _Hu, u_ ), we choose the circle _u_ 1 = 0,   = 1 to obtain λ2 = 2 = _ϕ_ ( _e_ 1) = min _ϕ_ { _v_ l).

**Theorem.** _Let_   _be the eigenvalues of an_ n × n _Hermitian matrix_ H. _Then for_ i < n

_Proof_. Define the function

The function _ϕ_ exists because it is the maximum of a continuous function on a compact set. According to Rayleigh's principle, if _u_ 1, . . ., _u_ n are mutually orthogonal unit eigenvectors belonging to λ1 . . ., λn,

Let _v_ 1, ..., _v_ i be given. We will find an admissible vector _u_ for which( _Hu, u_ )  . Consider all vectors _u_ 0 of the form

The conditions

take the form of _i_ linear homogeneous equations in _i_ \+ 1 unknowns _c_ 1,.. . ., _c_ i+1. Since there are more unknowns than equations, the homogeneous system (10) has a solution _c_ _j_ = λγ _j_ with not all γ _j_ = 0 and with λ arbitrary. By letting λ = (Σ |γ _j_ |2)−1/2 we obtain a solution _c_ _j_ of (10) for which Σ | _c_ _j_ |2 = 1. Then the vector _u_ 0 = Σ _c_ _j_ _u_ _j_ satisfies all the constraints in the maximum problem (7). Therefore, the _maximum_ value _ϕ_ satisfies the inequality

But

We have just shown that _ϕ_ ( _v_ l, . . ., _v i_)   for any set of vectors _v_ 1, . . ., _v i_. But (8) shows that equality is attained when _v_ 1 = _u_ 1,..., _v i_ = _u i_. Therefore, λ i+l is precisely the minimum of _ϕ_ , as the theorem asserts.

**E XAMPLE 2.** Courant's principle for λ,3 implies that

This is, indeed, evident because

while, for _any_ unit vector _u_ , the weighted average  .

#### **PROBLEMS**

**1.** Let _A_ be a Hermitian matrix with eigenvalues  . Let _B_ be a Hermitian matrix with eigenvalues  . Suppose that

Using Rayleigh's principle, prove that

By applying Rayleigh's principle for the lower eigenvalues, can you conclude that   Why not?

**2.** Under the conditions of Problem 1, use Courant's principle to conclude that

**3.** Let _A, B,_ and _C_ be Hermitian matrices, with eigenvalues _α j_, _α j_, _γ j_. Let _A_ = _B_ \+  _C_ Assume that _B_ and C are independent of  . Show that, for each _j_ = 1, . .., _n_ ,

**4.** In Problem 3, if _A_ and _B_ are not assumed to be Hermitian, show by a counterexample that the conclusion is false.

**5.** Let _A, B,_ and _P_ be Hermitian matrices. Let _P_ be positive definite, and let _A_ = _B_ \+ P. If _α_ 1 > · · · > _α_ n and _β_ 1 > · · · > _β_ n are the eigenvalues of _A_ and of _B_ , show that _α_ 1 > _β_ 1,..., _α_ n > _β_ n.

### **6.4 THE INCLUSION PRINCIPLE**

In the analysis of large systems the following result is often useful:

**Theorem 1.** _Let_ A = ( _a_ ij) _be an_ n × n _Hermitian matrix_ , _with_ n > 1. _Let_ B _be the_ (n − 1) × (n − 1) _Hermitian matrix formed by deleting the last row and the last column of_ A. _Let_   _be the eigenvalues of_ A, _and let_   _be the eigenvalues of_ B. _Then_ ,

**E XAMPLE 1.** Let

The eigenvalues   of _A_ satisfy the equation

Therefore,

But _β_ 1 = 1. Therefore,  .

_Proof of the Theorem_. This is one of the many applications of the minimax principle. First we make a simple observation: The quadratic form ( _By, y_ ) equals the quadratic form ( _Ax, x_ ) if ( _x, e n_) = _x_ _n_ = 0 and if _x_ _i_ = _y_ _i_ ( _i_ = 1, . . ., _n_ − 1). For then

Let us show that   if _i_ < n. For any _v_ 1, ..., _v_ _i_ -1 in _E n_ we have

By the notation _x_ ⊥ _v_ _j_ we mean ( _a_ , _v_ j) = 0. The inequality (3) holds because the requirement _x_ ⊥ _e n_, on the right-hand side of (3), places an extra restriction on _x_. Thus, if the tallest blue-eyed man in a class is six feet tall, the tallest man with blue eyes _and_ brown hair cannot be more than six feet tall.

We will use the notation _x_ → _y_ to mean that _y_ is formed from _x_ by deleting the last component of _x_. For example,

if _v_ _i_ → _w_ _j_ ( _j_ = 1, . . ., _i_ − 1). In fact,

if _and only if_ the truncated vectors _y_ , _w_ 1, . . ., _w_ _i_ −1 satisfy

Given the correspondences _v j_→ _w j_, both sides of the equation (4) are _functions_ of the vectors _v_ 1, . . ., _v_ _i_ −1. Identical functions have the same minimum. Therefore,

Of course, if _i_ = 1, no vectors _v_ or _w_ appear, and we replace (5) by

The inequality (3) states that one function of _v 1_, ... _v_ _i_ -1 is always equal to or greater than another function of _v_ 1,. . ., _v_ _i_ -1. The minima of these two functions must satisfy the same inequality. For example, cos   sin _θ_ implies that 2  . By Courant's theorem, the minimum of the left-hand side of (3) is _α i_. In (5) we showed that the minimum of the right-hand side of (3) is _β_ i. Therefore,  .

It remains to show that  . In (5) we showed that

where we define

But Courant's theorem states that

In the minimization (7) we fix _v_ _i_ = _e_ _n_ ; in the minimization (9) we do not prescribe _v_ _i_ , and there are more admissible candidates to minimize _ϕ_. Therefore, the minimum _β_ 1 in (7) is   the minimum _α_ _i_ +1 in (9). This completes the proof.

#### **PROBLEMS**

**1.** Let

According to the inclusion principle, what inequalities must the eigenvalues of _A_ satisfy?

**2.** Let _A_ be an _n_ × _n_ Hermitian matrix. Let _B_ be the Hermitian matrix formed by deleting row _k_ and column _k_ from _A_. Show that the eigenvalues of _A_ and the eigenvalues of _B_ satisfy the inequalities in formula (1).

**3.** Let _A_ be an _n_ × _n_ Hermitian matrix. Let _B_ be the Hermitian matrix formed by deleting the last two rows and the last two columns of _A_. If _A_ has eigenvalues  , and if _B_ has eigenvalues    , show that

**4.** Let _A_ be an _n_ × _n_ Hermitian matrix. Let _U_ be any _n_ × _n_ unitary matrix. Let _B_ be the Hermitian matrix formed by deleting the last row and the last column of _U_ * _AU_. Show that the eigenvalues of _A_ and the eigenvalues of _B_ satisfy the inequalities in formula (1). By a suitable choice of _U_ , deduce the result of Problem 2.

**5.** * Let _A_ be a real, symmetric _n_ × _n_ positive-definite matrix. Let _c_ be any nonzero column vector with _n_ real components. Let _E_ be the ellipsoid ( _Ax_ , _x_ ) = 1. Let _E_ ′ be the ellipsoid which is the intersection of _E_ with the plane ( _x_ , _c_ ) = 0. If the principal axes of _E_ and of _E_ ′ have lengths   and lengths  , show that   ( _j_ = 1, . . . , _n_ − 1). (First prove the result if _c_ = _e_ _n_. Then, by suitable rigid motion _x_ ′ = _Ux_ , prove the result for arbitrary _c_ ≠ 0.)

**6.** * In the notation of Section 4.8, consider the mass-spring system  . Let the fundamental frequencies of vibration be    . If the component _x_ _n_ is constrained to be 0, a reduced system results with fundamental frequencies  . Show that  . (First develop a Courant principle

for the generalized eigenvalue problem _Kx_ = λ _Mx_.)

### **6.5 A DETERMINANT CRITERION FOR POSITIVE DEFINITENESS**

We now give another application of the inclusion principle:

**Theorem 1.** _Let_ H = (hij) _be an_ n × n _Hermitian matrix. Then_ H _is positive definite if and only if_

_Proof_. Let _H_ _k_ be the Hermitian matrix formed from the first _k_ rows and the first _k_ columns of _H_. We will prove by induction that _H_ = _H_ _n_ is positive definite if and only if det _H_ _k_ > 0 ( _k_ = 1, . . . , _n_ ).

Suppose that _H_ is positive definite. Then all the eigenvalues of _H_ are positive, and the product of the eigenvalues = det _H_ is positive. But every _H_ _k_ is positive definite if _H_ is positive definite, since the quadratic form of _H_ _k_ equals the quadratic form ( _Hx_ , _x_ ) when _x_ _k_ +l = · · · = _x_ _n_ = 0. Therefore, det _H_ _k_ > 0 for all _k_.

Conversely, suppose that all det _H_ _i_ > 0 for all  . Then _H_ 1 = ( _h_ 11) is positive definite. Supposing that _H_ _k_ is positive definite for some _k_ < _n_ , we will prove that _H_ _k_ +l is positive definite. If   are the eigenvalues of _H_ _k_ +1, and if   are the eigenvalues of _H_ _k_ , the inclusion principle tells us that

But all _β_ _i_ are positive, since _H_ _k_ is positive definite. Therefore, all _α_ _i_ are positive except, perhaps, _α_ _k_ +1. But

Since all the numbers _α_ 1, . . . , _α_ _k_ , det _H_ _k_ +l are positive, we conclude that _α_ _k_ +1 > 0 and that _H_ _k_ +l is positive definite. At the end of the induction, we conclude that _H_ _n_ = _H_ is positive definite.

In this proof we have used the fact that a Hermitian matrix _A_ is positive definite if and only if its eigenvalues λ _i_ are positive. The reader will recall that this is so because

where _u_ 1, . . . , _u_ _n_ are orthogonal unit eigenvectors of _A_.

#### **PROBLEMS**

**1.** For which values of _μ_ is the matrix

positive definite?

**2.** For which values of _μ_ is the matrix

positive definite?

**3.** Let _H_ be an _n_ × _n_ Hermitian matrix. Let   be formed from the _last k_ rows and the _last k_ columns of _H_. Show that _H_ is positive definite if and only if det  .

### **6.6 DETERMINANTS AS VOLUMES; HADAMARD'S INEQUALITY**

In some applications, for example, in the Fredholm Theory of integral equations, we need to estimate the absolute value of a determinant. Let _A_ = ( _a_ _ij_ ) be an _n_ × _n_ matrix with columns _a_ 1, . . . , _a_ _n_. Let || _x_ || = (∑ | _x_ _i_ |2)1/2, as usual, represent the length of a vector _x_ in _E_ _n_. Hadamard proved:

**Theorem 1**

_with equality if and only if some_ aj = 0 or _all_ (aj, ak) = 0 _when_ j ≠ k.

**E XAMPLE 1**

**E XAMPLE 2.** For all real angles _θ_ and  ,

**E XAMPLE 3**

Hadamard's Inequality has a very simple geometrical interpretation. In the real vector space of two or three dimensions, | det _A_ | equals the area or volume of the parallelogram or parallelepiped whose sides are the columns of the matrix _A_. In two dimensions, Hadamard's Inequality states that _the parallelogram with sides_ a1 _and_ a2 _has area_ equal to or less than _the area_ ||a1|| · ||a2|| _of the rectangle whose sides have the same lengths_. In three dimensions, the volume of a parallelepiped with sides that have given lengths is maximized if the angles at the vertices are all 90 degrees.

_Proof of the Theorem_. Let _α_ _i_ = || _a_ _j_ || ( _j_ = 1, . . . , _n_ ). We may assume all _α_ _j_ > 0 because, if any _α_ _j_ = 0, both sides of (1) are zero. We now state a _maximum problem: Find an_ n × n _matrix_ B, _whose columns_ bj _have given positive lengths _α_ j, for which_ | det B | _is maximized_. This problem surely has a solution, since | det _B_ | is a continuous function of the _n_ 2 variables _b_ _ij_ _defined on the compact set_  .

Assume that _B_ solves the maximum problem. Expand det _B_ by column _j_ ; if _C_ = ( _c_ _ij_ ) is the matrix of signed cofactors of _B_ , then

We know that | det _B_ |   because the diagonal matrix diag ( _α_ 1, . . . , _α_ _n_ ) is one competitor in the maximum problem.

In Section (4.4) we proved that, if _x_ and _y_ are ≠ 0,

with equality if and only if _x_ is some scalar multiple of _y_. Apply this result to the nonzero vectors

We find

with equality _only_ if the _j_ th column of _B_ has the form

where _c_ _j_ is the _j_ th column of the cofactor matrix. Since _B_ solves the maximum problem, we _must_ have the form (6). Otherwise | det _B_ | could be increased by replacing _b_ _j_ by the vector  , where _μ_ _j_ = _α_ _j_ /|| _c_ _j_ ||. (Keep in mind that the _j_ th column _c_ _j_ of the cofactor matrix remains constant if the _j_ th column _b_ _j_ is varied.)

From the form (6) we shall conclude that the columns of _B_ are orthogonal. If _k_ ≠ _j_ we have

But the sum in (7) is the expansion by column _j_ of the determinant of the matrix formed from _B_ by replacing column _j_ by a duplicate of column _k_. Since a matrix with two identical columns has determinant zero, we have ( _b_ _k_ , _b_ _j_ ) = 0 if _k_ ≠ _j_.

Since _B_ has orthogonal columns with lengths _α_ _j_ , we may compute

But   because _B_ solves the maximum problem. Therefore,  . Equality holds only if _A_ is another solution of the maximum problem. By the reasoning in the preceding paragraph, a matrix which solves the maximum problem must have orthogonal columns. This completes the proof of Hadamard's Theorem.

_Why are determinants equal, in absolute value, to volumes_? For two reasons: (i) The equality

does hold if the columns are orthogonal or if the columns are linearly dependent; (ii) neither the volume nor the determinant changes if a multiple of _a_ _j_ is subtracted from _a_ _k_ ( _k_ ≠ _j_ ). We have already proved (i) in equation (8), which holds for any square matrix with orthogonal columns.

We illustrate (ii) for _n_ = 3. Assume that _a_ 1, _a_ 2, _a_ 3 are linearly independent, i.e., they are not all in one plane. The volume _v_ ( _a_ 1, _a_ 2, _a_ 3) may be computed as follows.

We begin with the parallelogram _P_ generated by _a_ 1 and _a_ 2 in Figure 6.1. The vector _a_ 2 has a component λ _a_ l which is a multiple of _a_ 1 and a component _b_ 2 perpendicular to _a_ 1. The area of _P_ equals the area of the rectangle generated by _a_ 1 and the perpendicular component _b_ 2. Therefore,

Figure 6.1

Similarly, the vector _a_ 3 (not illustrated in Figure 6.1) has a component _μa_ 1 \+ _vb_ 2 in the plane of _P_ and a component _b_ 3 perpendicular to the plane of _P_. Since _b_ 3 is the altitude perpendicular to the base parallelogram _P_ , we have

But the vectors _a_ 1, _b_ 2, _b_ 3 are orthogonal. If _B_ is the 3 × 3 matrix with these vectors as columns, we have

Since _B_ arises from _A_ by subtracting multiples of columns from other columns, det _B_ = det _A_. This completes the proof of (9) for _n_ = 3.

#### **PROBLEMS**

**1.** Obtain a second proof of Hadamard's inequality in the following way. Let the columns of _A_ be the vectors _a_ 1, . . . , _a_ _n_. If the columns are dependent, det _A_ = 0, so assume that the columns are independent. Form orthogonal vectors _b_ l, . . . , _b_ _n_ successively by the relations

with

If _B_ is the matrix with columns _b_ 1, . . . , _b_ _n_ , show that

**2.** Let _A_ = ( _a_ _ij_ ) ( _i_ , _j_ = 1, . . . , _n_ ). Prove that

When does equality hold?

**3.** Let _A_ = ( _a_ _ij_ ) ( _i_ , _j_ = 1, . . . , _n_ ). Let ρ _i_ > 0 ( _i_ = 1, . . . , _n_ ). Prove that

When does equality hold?

**4.** Let _A_ = ( _a_ _ij_ ) ( _i_ , _j_ = 1, . . . , _n_ ). Let _α_ = max | _a_ _ij_ | > 0. Prove that

By discussing the particular matrices

show that | det _A_ | = _α_ _n_ _n_ _n_ /2 is possible for indefinitely large values of _n_.

### **6.7 WEYL'S INEQUALITIES**

Let _H_ be a Hermitian matrix. Suppose that we wish to know the eigenvalues of _H_ , but that we do not know its exact components. We are given a Hermitian matrix, _M_ , whose components are approximations to the components of _H_. If _M_ = _H_ \+ _P_ , where _P_ is the error matrix, we wish to estimate the differences between the eigenvalues of _M_ and the corresponding eigenvalues of _H_.

**Theorem 1.** _Let_ M = H + P, _where_ M, H, _and_ P _are Hermitian. Let_ M _have eigenvalues_  ; _let_ H _have eigenvalues_  ; _and let_ P _have eigenvalues_  . _Then_

_In particular, if_ P _is positive definite, then μ_ 1 > _η_ (i = 1, . . . , n).

_Proof_. The minimax theorem states that

But ( _Mu_ , _u_ ) = ( _Hu_ , _u_ ) + ( _Pu_ , _u_ ). By Rayleigh's principle,

To find min ( _Pu_ , _u_ ), we observe that the eigenvalues of − _P_ are    . The minimum of any function is minus the maximum of minus the function. Therefore, by Rayleigh's principle for the greatest eigenvalue of − P,

From (3) and (4) we conclude

Therefore, if || _u_ || = 1,

For any given vectors _v_ 1, . . . , _v_ _i_ −1we conclude

If some functions satisfy an inequality, their minima satisfy the same inequality. Therefore, the minima, with respect to _v_ 1, . . . , _v_ _i_ −1, of the three functions in (7) satisfy the inequality

Here we have used the minimax principle for the eigenvalue _η_ _i_ of _H_ and for the eigenvalue _μ_ _i_ of _M_.

If _M_ is near _H_ , are the eigenvalues of _M_ near the eigenvalues of _H_? Let _M_ − _H_ = _P_. If the eigenvalues of _P_ are  , we need only to ask: If _P_ is "small," are its eigenvalues small? Define the _matrix norm_

Letting _u_ be the unit eigenvectors belonging to _ρ_ 1 and _ρ_ _n_ , we find

Along with Theorem 1, these inequalities establish Weyl's theorem:

**Theorem 2.** _Let_ M _and_ H _be Hermitian. Let_   _be the eigenvalues of_ M, _and let_   _be the eigenvalues of_ H. _Then_

_Proof_. By Theorem 1, if _P_ = _M_ − _H_ ,

But (10) implies

This completes the proof.

_How can we estimate_ || _P_ || = || _M_ − _H_ ||? From the definition (9) we have

But _P_ * = _P_. By Rayleigh's principle, max ( _P_ 2 _u_ , _u_ ) is the largest eigenvalue of _P_ 2. Since the eigenvalues of _P_ 2 are  , we conclude that

since  , or

**E STIMATE 1**

_Proof_

**E STIMATE 2**

_Proof_. Let _u_ be an eigenvector of _P_ belonging to _ρ_ = _ρ_ 1 or _ρ_ _n_. Let | _u_ _m_ | = max | _u_ _j_ | > 0. Then

**E XAMPLE 1.** Suppose that

According to Estimate 2, we have

Weyl's theorem now gives the inequalities

for the eigenvalues _μ_ _i_ of _M_ and the eigenvalues _η_ _i_ of _H_. If we use Estimate 1 instead, we find

**E XAMPLE 2.** _Weyl's inequalities do not hold for the class of non-Hermitian matrices_. Let _H_ be replaced by the non-Hermitian matrix

where   is a small error of measurement. We find

The eigenvalues of _A_ are all zero; the eigenvalues of _M_ are the three cube roots of  . Therefore, for _i_ = 1, 2, 3,

But, if _P_ = _M_ − _A_ ,

Therefore, || _P_ || = || _M_ − _A_ || = |   |, and from (21)

If the error in measurement is   = .001, the error in the computed eigenvalues is bigger by a factor of 100.

#### **PROBLEMS**

**1.** Suppose that _H_ is a Hermitian matrix, and suppose that some numerical method produces a unitary matrix _U_ for which

Estimate the eigenvalues of _H_.

**2.** Let _M_ and _H_ be Hermitian matrices, and suppose that  . Prove that the eigenvalues of _M_ and of _H_ satisfy:

**3.** Is one of the estimates in formulas (15) and (16) always equal to or greater than the other?

**4.** If _P_ = _P_ *, show that

### **6.8 GERSHGORIN'S THEOREM**

If _A_ is an _n_ × _n_ matrix which is strongly weighted along the main diagonal, the following estimate is useful:

**Theorem 1.** _Every eigenvalue_ λ _of an_ n × n _matrix_ A _satisfies at least one of the inequalities_

**E XAMPLE 1.** Let

Figure 6.2 shows the three Gershgorin circles; the shaded areas comprise all complex numbers λ satisfying at least one of the inequalities (1).

_Proof of Theorem 1_. Let _u_ be an eigenvector belonging to λ, and let | _u_ _m_ | = max | _u_ _j_ | for _j_ = 1, . . . , _n_. Since (λ _I_ − _A_ ) _u_ = 0,

Therefore,

Dividing (4) by | _u_ _m_ | > 0, we find

Therefore, λ lies in the _m_ th Gershgorin circle (1).

Figure 6.2 contains three circles, two of them overlapping. There are two _disjoint components D_ 1 and _D_ 2, where _D_ 1 is the upper circle   and where _D_ 2 is the union of the two lower circles. We will show that _D_ 1 contains 1 eigenvalue of _A_ and that _D_ 2 contains 2 eigenvalues of _A_.

**Figure 6.2**

**Theorem 2.** _Let_ D1, D2, . . . , Dk _be the disjoint components of the Gershgorin circles_ (1). _Let_ Di _be the union of_ ni _of the circles_ (1) (so that ∑ni = n). _Then_ Di _contains exactly_ ni _eigenvalues of_ A.

_Proof_ : This result follows from the fact (proved in Section 6.13) that the eigenvalues of a matrix are continuous functions of the components of the matrix. Define the matrix

We have

As _θ_ varies from 1 to 0, the _n_ eigenvalues of _A_ move continuously to _a_ 11, . . . , _a_ nn.

We apply Gershgorin's Theorem to each matrix _A_ ( _θ_ ) (observe Figure 6.2). As _θ_ shrinks from 1 to 0, the three Gershgorin circles shrink to the points 4 _i_ , 0, and 2. The eigenvalues in _D_ 1 for _θ_ = 1 must move to 4 _i_ as _θ_ → 0. The eigenvalues in _D_ 2 move either to 0 or to 2 as _θ_ → 0. But in the limit there must be _one_ eigenvalue at 4 _i_ , _one_ eigenvalue at 0, and _one_ eigenvalue at 2. Therefore, _D_ 1 must originally have contained one eigenvalue, and _D_ 2 must have contained two eigenvalues. As the three circles shrink, there is no way in which an eigenvalue originally in _D_ 1 can move _continuously_ to any point in _D_ 2. This reasoning, based on the specific example in Figure 6.2, entails no loss of generality, and the theorem is proved.

#### **PROBLEMS**

**1.** If _B_ is an _n_ × _n_ matrix for which

prove that det _B_ ≠ 0.

**2.** By using Theorem 2, show that the matrix

has at least two real eigenvalues.

**3.** Let _A_ be an _n_ × _n_ matrix with real components. Show that the imaginary parts of the eigenvalues of _A_ satisfy

**4.** Let _A_ be an _n_ × _n_ matrix. Let _ρ_ 1, . . . , _ρ_ _n_ be any _n_ positive numbers. Show that every eigenvalue, λ, of _A_ must satisfy at least one of the inequalities

**5.** What theorem results if Gershgorin's Theorem is applied to the transposen of _A_?

### **6.9 VECTOR NORMS AND THE RELATED MATRIX NORMS**

For many purposes we have found it desirable to have a measure of the size of a vector or of a matrix. If λ, is a scalar, i.e., a real or complex number, we let |λ| represent the familiar absolute value or modulus. Let _x_ , _y_ , . . . be vectors in the space of vectors with _n_ real components _or_ with _n_ real or complex components, where _n_ is a fixed integer   2. A _norm_ of a vector _x_ is often designated by the symbol || _x_ ||, but we shall usually prefer the less cumbersome symbol | _x_ |. We should like the norm to have the following properties:

There are positive constants _α_ and _β_ such that

We shall call any function | _x_ | of the vector _x_ which has all these properties a _regular vector-norm_.

**E XAMPLE 1.** The Euclidean norm is

We shall reserve the symbol || _x_ || for this particular norm, which clearly has properties (1) and (2). Property (3) follows from the Schwarz Inequality (Section 4.4) in this way

Continuity (4) follows from the continuity of   as a function of the scalar  . Finally, (5) holds with _α_ = 1 and  .

**E XAMPLE 2.** If _P_ is positive definite, define

This is the so-called _Riemannian metric_ , which is used in differential geometry and in the theory of relativity. For _P_ = _I_ we have the Euclidean norm. The norm (8) clearly has properties (1) and (2). To prove the triangle inequality for this norm, we will first show that

If _P_ has the eigenvalues λ _k_ > 0 and the unit orthogonal eigenvectors _u_ _k_ , and if

then

This proves (9). We can now obtain the triangle inequality:

Continuity (4) is obvious; and (5) holds because, if λ1 and λ _n_ are the greatest and least eigenvalues of _P_ , and if || _x_ || is the Euclidean norm (6), then

Since max   max | _x_ _k_ |, we have (5) with   and  .

**E XAMPLE 3.** Define

Here all properties (1)–(5) are obvious.

**E XAMPLE 4.** Define

For this norm, as well, properties (1)–(5) are obvious.

The requirements (1)–(5) are _not_ independent. We will show in Theorem 1 that every function | _x_ | with the properties (1), (2), (3) necessarily has the property of continuity (4) _and_ the property of comparability (5). We call (5) the property of _comparability_ because it shows that, if | _x_ | and | _x_ |′ are any two regular norms, then the ratio is bounded from above and below:

Thus, if | _x_ |′ = || _x_ ||, the Euclidean norm of Example 1, and if | _x_ | is the norm of Example 4, we have

**Theorem 1.** _Let_ |x| _be any function defined on the_ n- _dimensional vector space which has properties_ (1), (2), (3). _Then the function_ |x| _also has properties_ (4) _and_ (5).

_Proof_. Let

Then, by homogeneity (2) and by repeated application of the triangle inequality (3),

Therefore,

From this inequality we can prove continuity. Let _c_ be a fixed vector. We must show that

where

By the triangle inequality

and also

Therefore,

Therefore, to prove continuity (14), we need only to show that | _x_ | → 0 as _x_ → 0; but this is assured by the inequality   max | _x_ _k_ |, which we proved in (13).

To complete the proof of comparability (5), we consider the continuous function | _x_ | on the closed, bounded set _C_ of vectors for which max | _y_ _k_ | = 1. The set _C_ is illustrated in Figure 6.3 for _n_ = 2.

**Figure 6.3**

In general, _C_ is the boundary of the _n_ -dimensional unit hypercube. Define

This minimum is achieved at some point _y_ 0 on _C_. Then _α_ = | _y_ 0| > 0 because _y_ 0 ≠ 0; in fact, max  . We now have   for all _y_ on _C_. For any vector _x_ we may write _x_ = _μy_ where _μ_ = max | _x_ _k_ | and _y_ lies on _C_. Therefore,

This completes the proof of the theorem.

Given a regular vector-norm, | _x_ |, and a fixed _n_ × _n_ matrix, _A_ , we may measure the "size" of _A_ as the greatest value of the ratio | _Ax_ |/| _x_ | for all _x_ ≠ 0. Thus, the matrix _I_ has "size" 1, the matrix −7.3 _I_ has "size" 7.3, etc. The ratio | _Ax_ |/| _x_ | shows how much the transformation _A_ magnifies the norm of the vector _x_.

If | _x_ | is a regular vector-norm on the _n_ -dimensional vector space, and if _A_ is an _n_ × _n_ matrix, we define the _related matrix-norm_ as

For this definition to make sense, we must show that the maximum (19) exists. Let _S_ be the "unit sphere"

relative to the norm | _x_ |. For example, if _n_ = 2 and | _x_ | is denned as | _x_ 1| + | _x_ 2|, the set _S_ is the diamond shape in Figure 6.4.

**Figure 6.4**

The set _S_ is bounded because, by comparability (5), for | _y_ | = 1

Moreover, _S_ is closed because | _x_ | is continuous, and

If _x_ is any nonzero vector, we have _x_ = | _x_ | _y_ , where _y_ lies on _S_. Then

Therefore, the maximum (19) exists

because the function | _Ay_ | is a continuous function of _y_ defined on the closed, bounded set _S_ = { _y_ || _y_ | = 1}.

**E XAMPLE 5.** Let | _x_ | = || _x_ ||, the Euclidean norm of Example 1. The related matrix-norm is

But

By Rayleigh's Principle, the maximum value of this quotient is the largest eigenvalue λmax of the positive semidefinite Hermitian matrix _A_ * _A_. Taking square roots, we find

For instance, if

then λmax ( _A_ * _A_ ) = 4, and  .

**E XAMPLE 6.** Define | _x_ | = max | _x_ _k_ |. Then

But, if max | _y_ _j_ | = 1,

Conversely, let _m_ be an index such that

If, for this index _m_ ,

define the "unit vector" _y_ 0 by

For this vector _y_ we have

The inequality (25) states that

whereas (26) states that, for the particular "unit vector" _y_ 0

Therefore,

For instance, if

we have | _A_ | = 14 because |− 7| + |7| > |3| + |9|. The norm (27) is convenient because it can be readily calculated.

The next theorem will show that the matrix norm related to any regular vector norm, has many useful properties.

**Theorem 2.** _In the_ n- _dimensional space let_ |x| _be any regular vector norm_ , _i_. _e_., _any norm satisfying_ (1), (2), (3) _and hence_ (4) and (5). _Let_ |A| _be the related matrix norm_ (19) _for_ n × n _matrices_. _Then_

_Proof_. Property (i) holds because | _Ix_ |/| _x_ | = 1 for all _x_. We have   because | _A_ | is the maximum of numbers  . Since |0 _x_ | = |0| = 0 for all _x_ , we have |0| = 0. Conversely, suppose | _A_ | = 0. Then, by the definition (19), | _Ax_ | = 0 when _x_ is any one of the basic vectors _e_ 1, . . . , _e_ _n_. But this says that | _a_ _j_ | = 0 for all columns _a_ _j_ of _A_. But the vector norm | _a_ _j_ | = 0 only if _a_ _j_ = 0; therefore, all columns of _A_ are zero, and _A_ is the zero matrix. This proves (ii).

Property (iii) holds because |(λ _A_ ) _x_ | = |λ( _Ax_ )| = |λ|| _Ax_ |.

Property (iv) comes from the triangle inequality (3).

Let _y_ 0 be a vector with | _y_ 0| = 1 such that, as in (21),

Then

which proves (iv).

If _y_ 1 satisfies | _y_ 1 | = 1 and | _AB_ | = | ( _AB_ ) _y_ 1 |, then

which proves (v); here we have used (vi), which is an immediate consequence of the definition. We note parenthetically that | _A_ | _is the_ smallest _number_   _for which_   _for all_ x.

To prove continuity (vii) and (viii), we make a simple observation: _n_ × _n_ matrices _A_ may be regarded as vectors ( _a_ 11, _a_ 12, . . . , _a_ _nn_ ) in the space of _n_ 2 dimensions. Conversely, every vector _z_ with _n_ 2 components can be written as a matrix _A_ , where _z_ 1 = _a_ 11, _z_ 2 = _a_ 12, . . . , _z_ _n_ +1 = _a_ 21, . . . , _z_ _n_ 2 = _a_ _nn_ Properties (ii), (iii), (iv) show that the matrix norm | _A_ | establishes a _vector_ norm with properties (1), (2), (3) on the space of vectors _z_ with _n_ 2 components. Continuity (vii) and comparability (viii) now follow from Theorem 1.

_The product inequality._ In Theorem 2, which we have just proved, we would like to replace the inequality   by equality | _AB_ | = | _A_ || _B_ |. This would be analogous to the law | _ab_ | = | _a_ || _b_ | which holds for the moduli of complex numbers. Unfortunately, for   _there is no matrix norm with the properties_

To see this let _A_ = _B_ = any nonzero matrix whose square is zero, e.g., the matrix with 1 in component 2, 1 and with zeros elsewhere. In two dimensions,

Then (28) implies | _A_ | > 0, | _B_ | > 0, | _AB_ | = | _A_ || _B_ | > 0; but | _AB_ | = | 0 | = 0, which is a contradiction.

**Figure 6.5**

_Vector norms and convex bodies._ We shall show that all norms in a real, finite-dimensional vector space are characterized as follows: Let _K_ be a convex body (i.e., a bounded, closed, convex set) which is symmetric about the origin. By convex we mean that, if _x_ and _y_ lie in _K_ , the whole line segment (1 − _ϑ_ ) _x_ \+ _ϑy_ (0 < _ϑ_ < 1) lies in _K._ We call _K_ a _generalized unit sphere_ ; it will consist of all points _x_ for which  . Let || _z_ || represent ordinary Euclidean distance. To find the norm | _z_ | belonging to any vector _z_ ≠ 0, draw the ray from the origin through the point _z_ , as in Figure 6.5. This ray intersects the boundary of _K_ in a point _z_ ′. The norm of _z_ is the ratio

**Theorem 3.** _Let_ | _x_ | _be a regular vector norm. Let_

_Then_ K _is a closed, bounded, convex set which is symmetric about the origin and in which the origin is an interior point. Further, if_ z ≠ 0, _and_ z′ _is the boundary point of_ K _on the ray from the origin through_ z, _then_ | z | _is the ratio of the Euclidean lengths_ || z || _and_ || z′ ||.

_Proof_. By continuity (4), _K_ is closed: If _x_ _k_ → _x_ , and if all  , then  . _K_ is bounded by comparability (5):   implies max | _x_ _j_ |  . _K_ is convex because, if   and  , then

Let _z_ ≠ 0, and let _z_ ′ be the boundary point on the ray from the origin through _z_. First we show | _z_ ′ | = 1. We have   because _z_ ′ lies in the closed set _K_. Suppose | _z_ ′ | < 1. Let _D_ _ε_ be the open set of points _x_ for which

For all _x_ in _D_ _ε_ we have the norm

If we choose _ε_ = (1 − | _z_ ′ |)/ _β_ , then we have | _x_ | < 1 for all _x_ in _D_ _ε_. Then _z_ ′ would be an _interior_ point of _K_. This is a contradiction. Therefore, | _z_ ′ | = 1. Since _z_ and _z_ ′ lie on the same ray, we may write _z_ = λ _z_ ′, where λ > 0 since _z_ ≠ 0. By homogeneity (2) we have | _z_ | = | λ | | _z_ ′ | = λ. But || _z_ || = λ || _z_ ′ ||. Therefore, | _z_ | = λ = || _z_ ||/|| _z_ ′ ||. _K_ is symmetric since | _z_ | = | − _z_ |.

The reader will observe that the Euclidean norms || _z_ || and || _z_ ′ || play no special role in the characterization of | _z_ |. It is only the scalar λ > 0 for which _z_ = λ _z_ ′, which is important.

**Theorem 4.** _Let_ K _be a closed, bounded, convex set in the space of_ n _dimensions. Let_ x = 0 _be an interior point of_ K, _and let_ K _be symmetric about the origin. We define_ | 0 | = 0, _and if_ z ≠ 0, _we define_ | z | _by the ratio_ (29), _where_ z′ _is the boundary point of_ K _on the ray from the origin through_ z. _Then_ | z | _is a regular vector norm, i.e_., | z | _has properties_ (1)–(5).

_Proof_. By Theorem 1, we only need to demonstrate (1), (2), and (3). Properties (1) and (2) are obvious. We must prove the triangle inequality (3). If _x_ = 0 or if _y_ = 0, (3) is trivial. Assume _x_ ≠ 0 and _y_ ≠ 0. Write

and where _x_ ′ and _y_ ′ are boundary points of _K_. Since _K_ is supposed to be convex, the point

lies _in K_. Therefore, if _z_ ′ is the boundary point of _K_ on the ray from the origin through _z_ , the point _z_ lies between the origin and _z_ ′, and   so that

But   states, according to (31) and (32),

Since λ ≡ | _x_ | and _μ_ ≡ | _y_ |, the last inequality says:

This completes the proof.

#### **PROBLEMS**

**1.** Prove that the matrix norm related to | _x_ | = ∑ | _x_ _k_ | is

**2.** If _n_ = 2, show that | _x_ | = 3 | _x_ 1 | + 5 | _x_ 2 | is a regular vector norm by verifying all properties (1)–(5). Draw the "unit sphere"  .

**3.** What is the matrix norm related to the vector norm in the last problem?

**4.** * If | _x_ | is the norm ( _Px_ , _x_ )l/2, where _P_ is positive definite, express the related matrix norm, | _A_ |, in terms of a certain generalized eigenvalue.

**5.** Define the matrix norm | _A_ | = max | _a_ _ij_ |. If | _x_ | is the vector norm | _x_ | = max | _x_ _k_ |, which of the properties (i)–(viii) does the norm | _A_ | _not_ have?

**6.** * Let _K_ be a closed, bounded, convex set in which the origin is an interior point. Let _z_ ≠ 0. Show that the ray { _x_ | _x_ = λ _z_ , λ > 0} intersects the boundary of _K_ in one and _only_ one point _z_ ′.

**7.** Suppose that the earth is travelling on a simple Keplerian orbit about the sun. In what regular vector norm, analytically, is the earth travelling on the surface of the "unit sphere"?

**8.** Let _ρ_ ( _A_ ) be the maximum of the moduli of the eigenvalues of an _n_ × _n_ matrix _A_. If | _A_ | is the matrix norm related to any regular vector norm, show that  .

**9.** Let _H_ and _K_ be _n_ × _n_ Hermitian matrices. Define _ρ_ as in the last problem. Prove that   by identifying _ρ_ with some matrix norm related to a regular vector norm.

**10.** Let  . Using the method of the last problem, show that there are positive numbers _α_ and _β_ for which

for all _n_ × _n_ Hermitian matrices _H_ , where _α_ and _β_ are independent of _H_.

**11.** If we define

show that  , and show that  .

**12.** If | _x_ | is a regular vector norm defined on _E_ _n_ , and if _K_ is any _n_ × _n_ nonsingular matrix, show that | _x_ |′ = | _Kx_ | is also a regular vector norm. Show that | _A_ |′ = | _KAK_ −1 | is the related matrix-norm.

**13.** * Let ibea given _n_ × _n_ matrix, and let _ε_ > 0. Show that a regular vector norm can be denned on _E_ _n_ for which the related matrix norm satisfies the inequality | _A_ | < _ρ_ ( _A_ ) + _ε_. (METHOD: By the result of the last problem, | _x_ | = || _Kx_ || is a regular vector norm if _K_ −1 exists. If _A_ is similar to a diagonal matrix, Λ, pick _K_ so that _KAK_ −1 = Λ. If _A_ cannot be diagonalized, let _U_ * _AU_ = _R_ , a right-triangular matrix. Now choose an appropriate diagonal matrix _D_ so that _DRD_ −1 is "almost" diagonal. Then let _K_ = _UD_.)

### **6.10 THE CONDITION-NUMBER OF A MATRIX**

Let | _x_ | be a vector-norm defined as in the preceding section. For definiteness, the reader may think of

If _A_ is a nonsingular square matrix, we define the _condition-number_

This number has a simple interpretation. Let the unit sphere | _x_ | = 1 be mapped by the transformation _y_ = _Ax_ into some surface _S_. The condition number _γ_ ( _A_ ) is the ratio of the largest to the smallest distances from the origin to points on _S_. Thus,  . In fact,

if λ1, . . . , λ _n_ are the eigenvalues of _A_ arranged so that  . This follows from setting _u_ and _υ_ equal to eigenvectors belonging to λ1 and λ _n_.

To obtain a convenient representation for _γ_ ( _A_ ), we note from (2) that

By the definition given in the preceding section, the matrix-norm | _A_ | related to the vector-norm | _x_ |, is

For instance, the norm | _A_ | related to the maximum norm (1) was computed in Example 6 of the last section:

But what is the _minimum_ of | _Aυ_ | for | _υ_ | = 1? We have

and equality is attained for some vector ( _Aυ_ ). Therefore,

From (5) and (7) we now obtain the expression

The condition number comes into many sorts of error analyses. Here we shall give only one example. Suppose that we are solving _Ax_ = _b_ , where the data _A_ and _b_ are not known exactly. What is the effect of errors _δA_ and _δb_ on the solution? Let

Assume that _A_ and _A_ \+ _δA_ are nonsingular, and that _b_ ≠ 0. Define the error ratios

_We will estimate ξ as a function of α and β_. Equation (9) yields

But

whereas

Therefore,

Multiplication by | _A_ −1 | and division by | _x_ | yield

To work in the ratios _α_ and _β_ , we write

and (since _b_ = _Ax_ )

Hence

Both identities (14) and (15) contain the condition-number _γ_ = | _A_ −1 || _A_ |! Now (13) yields

Assuming that _γα_ < 1, we find

or, written out,

#### **PROBLEMS**

**1.** Let

If all of the components of _A_ and _b_ are changed by increments with absolute values  , give an upper bound for the increments | _δx_ 1 | and | _δx_ 2 | in the components of the solution to _Ax_ = _b_.

**2.** Let

Assume that   and  . Give an upper bound for | _δx_ |/| _x_ | if _Ax_ = _b_ and ( _A_ \+ _δA_ )( _x_ \+ _δx_ ) = ( _b_ \+ _δb_ ).

**3.** Let _A_ be Hermitian and positive definite, with eigenvalues    . Relative to the Euclidean norm, | _x_ | = (∑ | _x_ _υ_ |2)1/2, what is the condition-number of _A_ ?

**4.** Prove that  .

**5.** * If | _B_ |/| _A_ | = _θ_ < 1, prove that

**6.** Relative to the Euclidean norm, show that _γ_ ( _UA_ ) = _γ_ ( _AU_ ) = _γ_ ( _A_ ) if _U_ is any unitary matrix.

### **6.11 POSITIVE AND IRREDUCIBLE MATRICES**

In the numerical analysis of elliptic partial differential equations, in probability theory, and in physics and chemistry, we encounter matrices _A_ = ( _a_ _ij_ ) with all  . In the simplest case, all _a_ _ij_ are positive. The basic theorem of Perron states:

**Theorem 1.** _Let_ A = (aij) _be an_ n × n _matrix with all_ aij > 0. _Then_ A _has a positive eigenvalue ρ, of multiplicity one, with ρ_ > | λi | _for all other eigenvalues_ λi _of_ A. _The eigenvalue ρ has an eigenvector_ u _all of whose components are positive_.

**E XAMPLE 1.** If _n_ = 2, we have det (λ _I_ − _A_ ) = 0 for λ = _ρ_ , where

If all _a_ _ij_ > 0, this expression is real and positive.

_Proof_. By _x_ > 0 for a vector _x_ we shall mean that all _x_ _i_ > 0; by  , that all  . By _A_ > 0 we mean that all _a_ _ij_ > 0. If   but _x_ ≠ 0, define

_Thus, ϕ is the largest number such that_  . For instance, if

then

Therefore,  , and   is the largest number _v_ such that  .

We can consider the problem of maximizing _ϕ_ ( _x_ ) as a function of _x_. We will show that

and that the maximum is achieved for _x_ = the eigenvector _u_ > 0.

To show that max _ϕ_ ( _x_ ) exists, we will restrict the domain of _ϕ_ ( _x_ ) to a closed, bounded set _C_ on which _ϕ_ ( _x_ ) is continuous. By the definition (1), we see that _ϕ_ ( _x_ ) = _ϕ_ ( _μx_ ) for any scalar _μ_ > 0. Therefore, _ϕ_ ( _x_ ) = _ϕ_ ( _y_ ) for the vector _y_ = _μx_ such that ∑ _y_ _i_ = 1. Since the definition (1) appears to give trouble for small _x_ _i_ , we will make a further restriction. Since

we find, multiplying the inequality by _A_ > 0, that

Thus, for any nonzero  , we have

where _z_ lies in the set

The set _C_ is closed and bounded because it is a continuous map, _z_ = _Ay_ , of the closed, bounded set of vectors _y_ for which   and ∑ _y_ _i_ = 1. The components _z_ _i_ are bounded away from zero:

Therefore, the maximum exists:

Let the maximum _ρ_ be attained for _z_ = _u ε C_. Then, since _ρ_ = _ϕ_ ( _u_ ),

If _Au_ ≠ _ρu_ , then the difference   has at least one positive component. Therefore, since _every_ component of _A_ is positive,

In other words,

This inequality shows that _ρ_ is not the greatest number _ϕ_ = _ϕ_ ( _υ_ ) such that  . Therefore, _ρ_ < _ϕ_ ( _υ_ ), and _ρ_ < max _ϕ_ ( _x_ ). This contradiction shows that (5) should read

The number _ρ_ > 0 is, therefore, an eigenvalue of _A_. For any other real or complex eigenvalue λ, we have λ _c_ = _Ac_ , with _c_ ≠ 0, and

In other words,   if _x_ is the vector with components  . Therefore,

If for any index _i_ in (7) we have the strict inequality

then  , but multiplication by _A_ > 0 yields

Then  . Therefore, | λ | = _ρ_ only if equality holds for all _i_ in (7). Since all _a_ _ij_ are positive,

only if the complex components _c_ _j_ have the same argument:

where _θ_ is independent of _j_. Then λ _c_ = _Ac_ implies

Division by any | _c_ _i_ | ≠ 0 yields λ > 0, and therefore λ = _ρ_. Therefore, if λ ≠ _ρ_ , and if λ, is an eigenvalue of _A_ , | λ | < _ρ_.

It remains only to show that the eigenvalue _ρ_ is not a multiple eigenvalue. If Δ(λ) = det (λ _I_ − _A_ ), we must prove that the derivative Δ′( _ρ_ ) ≠ 0. Since _ρ_ is the largest root of Δ(λ) = 0, the derivative Δ′(λ) is positive for λ > _ρ_. We will show that Δ′( _ρ_ ) > 0.

Differentiation of the characteristic determinant gives

where Δ _k_ (λ) is the characteristic determinant of the matrix _A_ _k_ formed by deleting row _k_ and column _k_ from _A_. For example,

The identity (8) was proved in Section 1.8.

We will prove all Δ _k_ ( _ρ_ ) > 0. Consider Δ1(λ) == det (λ _I_ − _A_ 1). The matrix _A_ 1 has positive components. By what we have already shown, _A_ 1 has a positive eigenvalue _ρ_ 1 of maximum modulus, and there is a positive eigenvector _u_ 1. Let the _n_ − 1 components of _u_ 1 be called _u_ 2, _u_ 3, . . . , _u_ _n_. The equation _A_ 1 _u_ 1 = _ρ_ 1 _u_ 1 states

Define the _n_ -component vector _x_ by setting _x_ 1 = 0 and _x_ _i_ = _u_ _i_ for  . Then

with > for _i_ = 1, and with = for  . Since

multiplication of (11) by _A_ > 0 gives

This inequality shows that _ρ_ 1 < _ϕ_ ( _υ_ ). Since   for all _υ_ > 0, we conclude that _ρ_ 1 < _ρ_. Since _ρ_ 1 is the greatest positive root of Δ1(λ) = 0, we have Δ1( _ρ_ ) > 0. Similarly, Δ _k_ ( _ρ_ ) > 0 for _k_ > 1. Therefore, Δ′( _ρ_ ) = ∑Δ _k_ ( _ρ_ ) > 0, and we conclude that _ρ_ has multiplicity one as an eigenvalue of _A_.

_Irreducible matrices_. Suppose that we do not have all _a_ _ij_ > 0, but only  . Under a certain natural assumption, there is still a positive eigenvalue of maximum modulus.

We shall say that the _n_ × _n_ matrix _A_ is _irreducible_ if, for all pairs of indices _p_ ≠ _q_ , either _a_ _pq_ ≠ 0 or there is some set of indices _i_ 1, . . . , _i_ _r_ such that

where the indices _p_ , _i_ 1, . . . , _i_ _r_ , _q_ are all different. (Thus,  ). For example, if   is the resistance between two nodes, _i_ and _j_ , in an electrical network, the matrix _A_ is irreducible if there is a chain of resistors connecting every pair of nodes, _p_ and _q_. If _A_ is not irreducible, the corresponding resistance network consists of two or more unconnected parts.

**E XAMPLE 2.** The matrix

is irreducible, and  . The eigenvalues of _A_ are 1, _i_ , − 1, − _i_. There is a positive eigenvalue of maximum modulus, but the positive eigenvalue is not the only eigenvalue of maximum modulus.

**Theorem 2.** _Let_ A _be an_ n × n _irreducible matrix with all_  . _If_ λ1, . . . , λn _are the eigenvalues of_ A, _let ρ_ = max |λ _υ_ |. _Then ρ is an eigenvalue of_ A. _The eigenvalue ρ has multiplicity one_ , _and there is associated with ρ an eigenvector_ u _all of whose components are positive_.

_Proof_. First we will prove that

Consider the matrices _A_ , _A_ 2, . . . , _A_ _n_ −1. Let _p_ ≠ _q_. The matrix _A_ has a component _a_ _pq_. The matrix _A_ 2 has a component ∑( _i_ ) _a_ _pi_ _a_ _iq_ , which is positive if any single term _a_ _pi_ _a_ _iq_ is positive. The matrix _A_ _r_ +1 has a component

which is positive if any of its terms are positive. But the hypotheses of non-negativity and of irreducibility imply that at least one of the terms

is positive. Therefore, the _p_ , _q_ -component of the sum

is positive. (In fact, the _p_ , _q_ -component of any sum ∑ _α_ _υ_ _A_ _υ_ with positive coefficients, would be positive.) The diagonal terms of the sum (14) are also positive because the first term in the sum is _I_. Since the sum (14) equals ( _I_ \+ _A_ ) _n_ −1, the assertion (13) is proved.

If   but _x_ ≠ 0, we again define

The number _ϕ_ ( _x_ ) is the largest positive number _ϕ_ such that  . We will show that _ϕ_ ( _x_ ) attains a maximum value. If   and _x_ ≠ 0, define _y_ = _x_ /∑ _x_ _i_. By the homogeneity of the expression _ϕ_ ( _x_ ), we have _ϕ_ ( _x_ ) = _ϕ_ ( _y_ ). As _x_ ranges over all non-negative, nonzero vectors, _y_ ranges over the bounded, closed set

It is not clear that _ϕ_ ( _y_ ) is continuous on _C_ 1, since the factor   may become infinite in the expression  . We bypass this difficulty by defining the closed, bounded set of _positive_ vectors

All components _z_ 1 of a vector _z_ in _C_ are bounded away from zero. In fact, if _α_ is the smallest component of the positive matrix ( _I_ \+ _A_ ) _n_ −1, we have all  . For every _y_ in _C_ 1 we have

Multiplying the inequality (16) by the positive matrix ( _I_ \+ _A_ ) _n_ −1, we find

Since _ϕ_ ( _z_ ) is the greatest number _ϕ_ such that  , the inequality (17) implies that  .

For any vector  , with _x_ ≠ 0, we set up the correspondence _x_ → _y_ → _z_ →. Then

But the function _ϕ_ ( _z_ ) is _continuous_ on the closed, bounded set of positive vectors, _C_. Therefore, we have the existence of the maximum value of _ϕ_ ( _z_ ) for all _z_ in _C_. If the maximum is attained for _z_ = _u_ , the relation (18) implies

We will show that _ϕ_ ( _u_ ) is an eigenvalue of _A_ , and that _u_ is an eigenvector. We have

Unless equality holds in (20), multiplication by ( _I_ \+ _A_ ) _n_ −1 yields

But _Aυ_ > _ϕ_ ( _u_ ) _υ_ implies _ϕ_ ( _u_ ) < _ϕ_ ( _υ_ ), which contradicts the maximum property (19). Therefore, equality holds in (20).

If λ is any other eigenvalue of _A_ , let c ≠ 0, and

Then

Since some _a_ _ij_ may be zero, we cannot conclude that equality in (23) holds _only_ if all _c_ _j_ have the same argument. But we do have

Therefore,

Thus, _ϕ_ ( _u_ ) = _ρ_ ≡ max |λ _υ_ ( _A_ )|.

We have proved that _ρ_ is an eigenvalue of _A_ , and it remains only to show that _ρ_ is a _simple_ eigenvalue. If _ρ_ = λ1, λ2, . . . , λ _n_ are the eigenvalues of _A_ , then the eigenvalues of the _positive_ matrix ( _I_ \+ _A_ ) _n_ −1 are

But

Theorem 1, applied to the positive matrix ( _I_ \+ _A_ ) _n_ −1, implies that (1 + _p_ ) _n_ −1 is a simple eigenvalue of ( _I_ \+ _A_ ) _n_ −1. Therefore, _ρ_ is a simple eigenvalue of _A_ , and the proof is complete.

#### **PROBLEMS**

**1.** Let _c_ 1, . . ., _c_ _n_ be complex numbers. Let _a_ _i_ 1, .. ., _a_ _in_ be positive. Prove that

only if, for some argument _θ_ independent of _j_ ,

**2.** Let

Let _ρ_ ( _A_ ) be the maximum of the absolute values of the eigenvalues of _A_ , and let _ρ_ ( _A_ +) be defined similarly for _A_ +. Prove that  .

**3.** Let

Using _x_ , find a lower bound for _ρ_ ( _A_ ).

**4.** * Let _A_ > 0. For any  , with _x_ ≠ 0, define

Prove that the minimum value of _ψ_ ( _x_ ) exists and equals _ρ_ ( _A_ ).

**5.** Using Theorem 1 and the result of Problem 4, show that, if _A_ > 0,

for all nonzero  . Apply this result to obtain a two-sided inequality for _ρ_ ( _A_ ) in Problem 3.

**6.** * Let _A_ be an irreducible _n_ × _n_ matrix for which

Prove that det _A_ ≠ 0 if the > sign holds for at least one _i_.

**7.** Prove that the _n_ × _n_ matrix _A_ is reducible, i.e., not irreducible, if and nly if there is a proper subset _S_ of the integers 1, . . ., _n_ such that every system of equations _Ax_ = _b_ contains the smaller system of equations

For example, if _A_ is the reducible matrix

every system _Ax_ = _b_ contains the smaller system

In this example, _S_ = {1, 3}.

**8.** A permutation matrix _P_ is a matrix such that _Px_ = _y_ rearranges the components of _x_. Thus, the rearrangement _y_ 1 = _x_ 2, _y_ 2 = _x_ 3, _y_ 3 = _x_ 1 is accomplished by

A permutation matrix is unitary; its columns consist of a rearrangement of the columns of the identity matrix. Using the result of Problem 7, show that an _n_ × _n_ matrix _A_ is reducible if and only if there is a permutation matrix _P_ such that _PAP_ _T_ has the form

where _B_ is an _m_ × _m_ matrix with  . What is the matrix _P_ for the example in Problem 7?

**9.** Let _A_ be a Markov matrix: The component _a_ _ij_ < 0 represents the probability of transition from state _i_ to state _j_. Thus, Σ( _j_ ) _a_ _ij_ = 1 for all _i_. Prove that _A_ has the eigenvalue λ1 = 1 with multiplicity one, and that |λ _υ_ | < 1 for all other eigenvalues of _A_. Deduce the same conclusions for _A_ _T_. Prove that there is a row-vector _u_ _T_ > 0, with Σ _u_ _i_ = 1, which _u_ _T_ _A_ = _u_ _T_.

**10.** * In the notation of Problem 9, let x1   and let Σ _x_ _i_ = 1 (The component _x_ _i_ represents the initial probability of state _i_ ). Prove that

[Develop _x_ = col ( _x_ 1, . . ., _x_ n) in principal vectors of _A_ _T_. The number _u_ _i_ gives the probability of state _i_ after infinitely many transitions.]

**11.** * In Problem 2, show that _p_ ( _A_ ) = _p_ ( _A_ +) if and only if the components _a_ _μv_ have the form

### **6.12 PERTURBATIONS OF THE SPECTRUM**

Let _A_ be an _n_ × _n_ matrix with eigenvalues λ1, . . ., λ _n_ and with corresponding eigenvectors _u_ 1, . . ., _u_ _n_. A small change _δA_ in the matrix produces changes _δ_ λ _i_ in the eigenvalues and changes _δu_ _i_ in the eigenvectors. The purpose of this section is to show approximately how the eigenvalues and the eigenvectors change as the matrix changes. We shall treat only the easiest case, in which the eigenvalues λ _i_ are distinct.

According to Section 4.1, the eigenvectors _u_ _j_ are linearly independent and are unique, except for nonzero scalar multiples _αu_ _j_. We have _Au_ _j_ = λ _j_ _u_ _j_ and

In this equation we consider

We must be careful how we define _δu_ _j_. If _δA_ = 0, then _δ_ λ _j_ = 0 ( _j_ = 1, . . ., _n_ ); but the perturbation equation (1) is satisfied by any _δu_ _j_ which is a multiple of _u_ _j_. To insure _δu_ _j_ = 0 if _δA_ = 0, we shall normalize the perturbed eigenvector by the assumption that, in the expansion

the coefficient ( ) of _u_ _j_ remains equal to 1 when _A_ is replaced by _A_ \+ _δA_. In other words, we shall require expansions

_The unknowns are now δ_ λj _and the coefficients ε_ jk _for_ j ≠ k.

If the components of the matrix _δA_ are very small, equation (1) becomes, to first order,

where the neglected terms, ( _δA_ ) _δu_ _j_ and ( _δ_ λ _j_ ) _δu_ _j_ are of second order. Since _Au_ _j_ = λ _j_ _u_ _j_ , equation (4) yields

To compute the unknowns _δu_ _j_ and _δ_ λ _j_ we will use the _principle of biorthogonality_ :

_Let_ u1, . . ., un be eigenvectors corresponding to the eigenvalues λ1, . . ., λn _of an_ n × n _matrix_ A. _Assume_ λ _i_ ≠ λj _for_ i ≠ j. _Let_ v1, . . ., vn _be eigenvectors corresponding to the eigenvalues_   of A*. _Then_

This generalizes the orthogonality which holds for the eigenvectors _u_ _i_ = _v_ _i_ of Hermitian matrices _A_ = _A_ *. To prove (6), we first observe that

But this is the complex conjugate of

Thus, _A_ * does have the _n_ distinct eigenvalues  .

For _i_ ≠ _j_ we write

Taking inner products, we find

Since ( _Au_ , _v_ ) = ( _u_ , _A_ * _v_ ) for all _u_ and _v_ , we have

or

Since λ _i_ ≠ λ _j_ , we conclude that ( _u_ _i_ , _v_ _j_ ) = 0.

The inequality ( _u_ _i_ , _v_ _j_ ) ≠ 0 now follows because, if _u_ _i_ were orthogonal to _all_ the _v_ 's, then a representation _u_ _i_ = Σ( ) _v_ _k_ would yield ( _u_ _i_ , _u_ _i_ ) = 0, and hence _u_ _i_ = 0.

To solve equation (5) for _δ_ λ _j_ , and _δu_ _j_ we will use the eigenvectors _v_ 1, . . ., _v_ _n_ of _A_ *. By the normalization (3), the perturbation _δu_ _j_ is a combination of _u_ _k_ for _k_ ≠ _j_. Therefore, ( _δu_ _j_ , _v_ _j_ ) = 0. Now (5) yields

But

Therefore, since ( _u_ _j_ , _v_ _j_ ) ≠ 0,

To find _δu_ _j_ take the inner product of equation (5) with _v_ _k_ for _k_ ≠ _j_ :

since

But the normalization (3) gives

Equations(9) and (10) now give the required coefficients

**E XAMPLE.** Let _A_ = diag (λ1, . . ., λn), where λ _i_ ≠ λ _j_ for _i_ ≠ _j_. Let ( _δA_ ) = _ε_ B, where B = (b _ij_ and _ε_ is a small parameter. In this case, we may take

for the eigenvectors of _A_ and of _A_ *. Formula (8) now yields

Formula (11) gives

Now (3) gives

In other words, _δu_ _j_ is the vector whose _j_ th component is 0 and whose _k_ th component ( _k_ ≠ _j_ ) is _εb_ _kj_ /(λ _j_ −λ _k_ ).

#### **PROBLEMS**

**1.** Verify the principle of biorthogonality for the matrix

**2.** Let _A_ be defined as in Problem 1. If

compute the variations _δ_ λ1, _δ_ λ2, _δu_ 1, _δu_ 2,

**3.** * Let _A_ 0 be an _n_ × _n_ matrix with distinct eigenvalues λ10, . . ., λ _n_ 0 and corresponding eigenvectors _u_ 10, . . ., _u_ _n_ 0. Let _v_ 10, . . ., _v_ _n_ 0 be eigenvectors corresponding to the eigenvalues  . For small | _ε_ | let

For _j_ = 1, . . ., _n_ , let _A_ have the eigenvalues and eigenvectors

Normalize _u_ _j_ by the requirement

Devise a recursive scheme for computing the quantities

**4.** For small | _ε_ | let

Does _A_ have an eigenvalue which can be developed in a convergent power series of the form

**5.** * Let _A_ ( _ε_ ) be an _n_ × _n_ matrix whose components are analytic functions of _ε_ in the neighborhood of _ε_ = 0. Let _A_ (0) have distinct eigenvalues λ10, . . ., λ _n_ 0. Let

If

are the coefficients _c_ 1( _ε_ ), . . ., _c_ n( _ε_ ) analytic functions of _ε_? Using Cauchy-integral representations

show that _A_ ( _ε_ ) has distinct eigenvalues which are analytic functions of _ε_ in the neighborhood of _ε_ = 0.

**6.** * Using the assumptions in Problem 5, show that the function λ _j_ ( _ε_ ) is the solution of the analytic differential equation

which satisfies the initial condition λ(0) = λ _j_ 0.

**7.** * Let _A_ ( _ε_ ) be an _n_ × _n_ matrix whose components are analytic functions of _ε_. For _ε_ = 0 let _A_ have the simple eigenvalue λ0, and let

For _ε_ in the neighborhood of zero, let λ( _ε_ ) be a simple eigenvalue of _A_ , where λ( _ε_ ) is analytic and λ(0) = λ0. For _ε_ near zero show that the equations

determine a _unique_ vector _u_ ( _ε_ ) with _analytic_ components, and with _u_ (0) = _u_ 0 {Show that rank [ _A_ ( _ε_ ) − λ( _ε_ ) _I_ ] = _n_ − 1, and show that the row-vector ( _v_ 0)* is _not_ a linear combination of the rows of _A_ ( _ε_ ) − λ( _ε_ ) _I_. Hence, show that _u_ ( _ε_ ) is the solution of an equation _B_ ( _ε_ ) _u_ ( _ε_ ) = _c_ , where _B_ ( _ε_ ) is an _n_ × _n_ nonsingular matrix formed from _n_ − 1 rows of _A_ ( _ε_ ) − λ( _ε_ ) _I_ and the row ( _v_ 0)*}.

### **6.13 CONTINUOUS DEPENDENCE OF EIGENVALUES ON MATRICES**

A frequently used but seldom proved fact in matrix theory is that the eigenvalues λ1,...,λ _n_ of a matrix _B_ = ( _b ij_) depend continuously on the coefficients b11, b12, . . ., b _nn_.

**Theorem 1.** _Let µ_ 1, . . ., _µ_ s _be the different eigenvalues of an_ n × n matrix A = (aij). _Let the eigenvalue µ_ j _have multiplicity_ mj, _where_ Σ mj = n. _Then for all sufficiently small_   > 0 _there is a number_ δ = δ( ) > 0 _such that, if_   _for_ i, j = 1, . . ., n, _then the matrix_ **B** = (b _ij_ ) _has exactly_ mj _eigenvalues in the circle_ | λ − _μ j_| <   _for each_ j = 1, . . ., s.

**E XAMPLE 1.** Let _n_ = 8. Suppose s = 4, _m_ 1 = 3, _m_ 2 = 2, _m_ 3 = 2, _m_ 4= 1. In other words, an 8 × 8 matrix _A_ has the different eigenvalues _μ_ 1, _μ_ 2, _μ_ 3, _μ_ 4, with multiplicities 3, 2, 2, 1. Draw the four circles of radiuse   > 0 about the roots _μ i_, as in Figure 6.6. The theorem states that, if   is less than some critical value  0 > 0, then there is a number δ > 0 such that, if | _b ij_ − _a ij_| < δ ( _i, j_ = 1, ..., 8), then _B_ has three eigenvalues in the first circle (counted by their multiplicities), two eigenvalues in the second circle, two in the third, and one in the last.

**Figure 6.6**

_Proof of the Theorem_. The proof uses the theory of functions of a complex variable. Let _B_ be a variable _n_ × _n_ matrix. Define

Let

Choose any positive   <  0. Then the circles

are nonintersecting. Define

All _ρ j_ are positive because the roots of   are the centers of the circles. The minima (4) are defined because   depends continuously on _λ_.

Because the determinant   depends continuously on all 1 + _n 2_ variables λ, _b_ 11,. . ., _b nn_, there is some δ > 0 such that

if   for all _i_ , _j_ = 1, . . ., _n_. Now consider the integral

This is the integral which counts the number _n j_ of roots _λ_ of the equation   which lie inside the circle _C j_. For instance, we are given that

By (5) the integrand in (6) is a continuous function of _λ_ , _b_ 11, ..., _b nn_ in the closed set

Therefore, the integral _n j_( _B_ ) is a continuous function of the _b ij_ satisfying   Since the integral cannot jump continuously from one integer to another, we must have

for all matrices _B_ with  . This completes the proof.

The reader will recognize that this is not really a theorem about matrices, but about polynomials. Since the coefficients of the characteristic polynomial   depend continuously on the variables _b_ 11,..., _b nn_ it was only necessary to prove that the roots of a polynomial depend continuously on the coefficients of the polynomial.

#### **PROBLEMS**

**1.** By a trivial modification of the preceding proof, show that the roots of the equation _λ_ n \+ _c_ 1 _λ_ _n_ −1 \+ ... + _c_ _n_ −1 _λ_ \+ _c n_ = 0 depend continuously on the complex variables _c_ 1, ... _c n_.

**2.** Let an _n_ × _n_ matrix _A_ have distinct eigenvalues _λ_ 1, ..., _λ_ Let _B_ be a matrix whose coefficients _b ij_ ( ) are analytic functions of   for |   | < _ρ_ , with _b ij_ (0) = _a ij_. Prove that, for sufficiently small |   |, _B_ has distinct eigenvalues _μ_ 1( ),..., _μ n_( ) which are analytic functions of   such that _μ j_(0) = λ _j_ (See Problem 5, Section 6.11.)

**3.** Let _B_ ( _θ_ ) be a real _n_ × _n_ matrix depending continuously on a parameter _θ_. Suppose that for _θ_ = 0 the matrix has _n_ distinct real eigenvalues, but that for _θ_ = 1 the matrix has some complex, nonreal eigenvalues. Show that _B_ ( _θ_ ) cannot have distinct eigenvalues for all _θ_ in the range 0 < _θ_ < 1.

* * *

 See, for example, R. S. Varga, _Matrix Iterative Analysis_. Englewood Cliffs, N. J.: Prentice-Hall, Inc., 1962.

## **7 NUMERICAL METHODS**

### **7.1 INTRODUCTION**

The preceding chapters have presented an introduction to the _theory_ of matrices. Computational procedures are surprisingly different from theory. For example, suppose that we wish to obtain a _numerical_ solution for a given set of 50 linear equations in 50 unknowns.

Shall we use Cramer's rule? Each _x i_ is expressed as the quotient of two 50 × 50 determinants. Expanding the denominator, det _A_ , by the first row, we find

Each cofactor ± det A1 _j_ is a 49 × 49 determinant. If the cofactors are known, formula (2) exhibits 50 multiplications. But the 49 × 49 cofactors must be expanded, and so on. The complete expansion of det _A_ , therefore, requires 50 × 49 × 48 × ... × 2 × 1 multiplications. This process would take _years_ on a modern electronic computer.

There _is_ a better way. A 50 × 50 system (1) can be solved in _seconds_ , as this chapter will show.

Why, then, should one have bothered to learn the theory? Theory is understanding. Numerical methods, themselves, cannot be understood or applied effectively without the underlying theory. And for applications we want, most of all, insight. We want qualitative information, which no amount of blind computation can provide. For example, the theory of the Jordan form shows that a linear system of differential equations _x′_ = _Ax_ may have solutions whose components behave like _t_ sin _ωt_ as _t_ → ∞. Computation would never reveal this. We shall even show that the Jordan form is numerically unstable; for most matrices whose eigenvectors do not form a basis, the Jordan form cannot be computed by _any_ method because of rounding error.

The literature on matrix computations is enormous. This book is not a survey, but a text. We shall study not all methods, but only a few of the best. For certain particular classes of matrices there are better techniques. But the methods presented here are among the fastest and the most reliable for the great bulk of problems to be met in practice.

### **7.2 THE METHOD OF ELIMINATION**

We wish to solve a system of equations _Ax_ = _b_ where _A_ = ( _a ij_) is an _n_ × _n_ matrix. _Ιƒ_ A _is a triangular matrix with all_ aii ≠ 0, _the system can be solved by recursion_. Suppose _a ij_ = 0 if _i_ > _j_. We first solve the last equation for _x n_, then the next-to-the-last equation for _x_ _n_ -1, etc.

**E XAMPLE 1.** To solve

we compute, successively,

_If_ A _is not a triangular matrix, we transform_ A _into a triangular matrix by operations which preserve the solution_ x.

**E XAMPLE 2.** To solve

first interchange the first two equations.

Then subtract twice the first equation from the last equation.

If we now add the second equation to the third equation, we obtain the triangular system in Example 1.

The reduction of a system of equations to triangular form can be accomplished by matrix manipulations. It is not necessary to write the letters _x_ 1, _x_ 2 etc.

**E XAMPLE 3.** To solve the system _Ax_ = _b_ in Example 2, write the 3 × 4 augmented matrix [ _A, b_ ]

We use this matrix to _represent_ the system _Ax = b._ First interchange the first two rows.

Subtract twice the first row from the last row.

If we now add the second row to the third row, we obtain

which is the augmented matrix  _T, c_ ] representing the triangular system _Tx_ = _c_ in [Example 1.

In Example 2 we used two operations: (a) interchanging two rows and (b) adding a multiple of one row to another. _If a rectangular matrix_ X _can be transformed into a matrix_ Y _by a succession of these operations, we shall write_ X ∼ Y. Evidently, _X_ ∼ _Y_ implies _Y_ ∼ _X;_ and { _X_ ∼ _Y_ and _Y_ ∼ _Z_ } implies _X_ ∼ _Z;_ and _X_ ∼ _X._

**Theorem 1.** _Let_ A _be an_ n × n _matrix. Then_ A ∼ T _for some triangular matrix_ T, _with_ tij = 0 _for_ i > j. _Moreover,_

_where_ R _is the number of row-interchanges used in transforming_ A _into_ T.

_Proof._ Note that we do not assume det _A_ ≠ 0. The theorem is obvious for _n_ = 1, with _A_ = _T_ = a11. If _n_ > 1, we look at the first column of A. If some element in the first column is nonzero, we can, if necessary, perform a row-interchange to obtain a matrix with a nonzero element in the 1, 1 position. Now, subtracting multiples of the first row from the other rows, we obtain

where _A′_ is an ( _n_ − 1) × ( _n_ − 1) matrix. If the first column of _A_ is zero, _A_ already has the form of the right-hand side of (7), with _t_ 11 = 0.

By induction, _A′ ∼_ some triangular matrix, say

Therefore, as (7) shows, A ∼ ( _t ij_) ( _i, j_ = l, . . . , _n_ ). The formula (6) for det _A_ follows because each row-interchange reverses the sign of the determinant.

**Theorem 2.** _Let_ A _be an_ n × n _matrix. Let_ B _be an_ n × k _matrix. Then there is a triangular matrix_ T _such that_ [A, B] ∼ [T, C], _where the systems_ AX = B _and_ TX = C _have the same solution or solutions_ X, _unless both systems have no solution._

_Proof._ By Theorem 1 there is a triangular matrix _T_ such that _A_ ∼ _T_. The transformation of _A_ into _T_ is achieved by a sequence of (a) row interchanges and (b) additions of multiples of rows to other rows. If the same sequence of operations is performed on the _n_ × ( _n_ \+ _k_ ) matrix [ _A_ , _B_ ], we obtain an _n_ × _k_ matrix _C_ for which [ _A_ , _B_ ] ∼ [ _T_ , _C_ ]. The systems _AX_ = _B_ and _TX_ = _C_ have the same solution or solutions _X_ because every operation (a) corresponds to interchanging two equations, and every operation (b) corresponds to adding a multiple of one equation to another equation.

In Example 2, _B_ was a vector. _To compute the inverse_ of A _if_ det A ≠ 0, _we let_ B = I because then we are solving the equation _AX_ = _I_. If all _t ii_ ≠ 0, the triangular system _TX_ = _C_ , where _C_ is an _n_ × _k_ matrix, is solved by solving the equation _Tx j_ = _c j_ for each column-vector _x j_ ( _j_ = 1, . . . , _k_ ) by recursion, as in Example 1.

COUNT OF OPERATIONS

To determine the speed of computation we need to know how many calculations of various sorts are made. Consider the reduction [ _A, B_ ] ∼ [ _T, C_ ], where _A_ is an _n_ × _n_ matrix and _B_ is an _n_ × _k_ matrix. Let

Evidently, _M_ 1 = 0 and _A_ 1 = 0 for any  , since a 1 × 1 matrix _A_ is already triangular. If _n_ > 1, consider the first column reduction. If the first column of _A_ is not zero, usually a row-interchange is made to bring the element of largest absolute value in the first column into the 1, 1 position. Again call the new matrix [ _A, B_ ]. We must now subtract a multiple of the row

from each row

to produce a row of the form

Explicitly, if _r i_ = _a i1_/ _a_ 11,

There is one division to produce _r i_. Formula (10) requires _n_ − 1 + _k_ multiplications and an equal number of subtractions. Since _n_ − 1 rows (9) are formed, the first column reduction can be achieved with

The first column now has zeros below the 1, 1 position. We now have to reduce the ( _n_ − 1) × ( _n_ − 1 + _k_ ) matrix below the first row and to the right of the first column. From formula (11) we obtain the recursion

Therefore, since _M_ 1 = _A_ 1 = 0, we have for

One readily verifies by induction:

Therefore, since   and  ,

Now we will count the number _m n_ of multiplications or divisions and the number _a n_ of additions or subtractions needed to solve the triangular system _TX_ = _G._ Assume all _t ii_ ≠ 0. Let  . Suppose that we have already solved for _x nj_, _x_ _n_ −1, _j_ , . . . , _x_ 2, _j_ for each column _j_ = 1, . . . , _k_. Only _x_ 1 _j_ ( _j_ = 1, . . . , _k_ ) remain to be calculated. So far, _m_ _n_ −1 and _a_ _n_ −1 operations have been used, since the triangular system _below_ the first equation of _TX_ = _C_ has been solved. Now

For each _j =_ 1, . . . , _k_ formula (15) requires _n_ − 1 multiplications, one division, and _n_ − 1 subtractions. Therefore, for  ,

Since _m_ 1 = _k_ and _a_ 1 = 0, we have

The next theorem summarizes our results:

**Theorem 3.** _Let_ A _be an_ n × n _matrix with_ det A ≠ 0. _If_ B _is an_ n × k _matrix, then the equation_ AX = B _can be solved by the method of elimination with_

and

_Proof._ From (14) and (17) we find (18) as the sum _M n_ \+ _m n_, and we find (19) as the sum _A n_ \+ _a n_.

Observe that _μ n_ and a _α n_ are each less than _n_ 2 _k_ \+ _n_ 3/3. For large _n,_ if _k_ = 1, _μ n_ and _α n_ are approximately _n_ 3/3. If _k_ = _n_ , _μ n_ and _α n_ are approximately 4 _n_ 3/3.

In the preceding discussion we reduced the original system _AX_ = _B_ to a triangular system _TX_ = _C_. The next example will show that, if all _t ii_ ≠ 0, the triangular system can be solved by matrix manipulations. For this purpose we use one additional operation: (c) dividing (or multiplying) a row by a nonzero scalar. This corresponds to dividing (or multiplying) equations by a nonzero scalar.

**E XAMPLE 4.** We will solve the triangular system

First we write down the augmented matrix [ _T, C_ ]:

By the elementary row-operations (a), (b), and (c) we will reduce this matrix to the form [ _I, R_ ]. In other words, we will reduce the system _TX_ = _C_ to the system _IX_ = _R_ , which will givethe solution _X_ = _R_. Division of the last row by _t_ 33 = −2 yields

We now add appropriate multiples of the last row to the other rows to obtain

Then we divide the second row by _t_ 22 = 4. (In computer programming, we take care not to waste time dividing elements which we know already equal zero.) We obtain

Subtraction of the second row from the upper row yields

Thus, we have found the solution

PIVOTAL STRATEGY

In practical digital computation, it is important to reduce cumulative rounding error. The following procedure has been found useful. In the reduction to triangular form, to produce zeros below the diagonal in a certain column, we first look for an element of maximum absolute value on or below the diagonal in that column. If this so-called "pivotal element" lies below the diagonal, we first perform a row-interchange to bring the pivotal element onto the diagonal. Then we subtract multiples of the pivotal row to produce zeros below the diagonal in the pivotal column. For example, to reduce the second column of the matrix

we first recognize the pivotal element −10. We interchange rows to obtain

We now subtract multiples of the pivotal row to obtain

Note that the multipliers used (in this case .4 and .1) lie between −1 and +1, since the pivotal element has maximum absolute value. In our example, the next pivotal element will be 4.1 because |14.1| > |3.4|.

#### **PROBLEMS**

**1.** * In the matrix-form of the solution of triangular systems, as illustrated by Example 4, verify the recursion formulas (16) for the numbers of operations, _m n_ and _a n_.

**2.** Solve _AX_ = _B_ if

**3.** Find the inverse of the matrix _A_ in Problem 2. (Here we are solving _AX_ = _I_. Begin by writing the augmented matrix [ _A,I_ ].)

**4.** By reduction to a triangular matrix _T_ , calculate the determinant of the matrix

Use only the elementary operations (a) and (b) to form _T_.

**5.** * If _A_ is an _n_ × _n_ matrix with det _A_ ≠ 0, and if _C_ is an _n_ × _n_ matrix, Theorem 3 shows that _AX_ = _C_ can be solved by the method of elimination with about   multiplications. Show that, in the particular case _C_ = _I_ , this number may be reduced to _n_ 3. Thus, _the inverse matrix can be formed with only_ n3 _multiplications_.

### **7.3 FACTORIZATION BY TRIANGULAR MATRICES**

In the preceding section we described a form of the method of elimination which applies to any system _Ax_ = _b_ in which det _A_ ≠ 0. For a wide class of matrices it is possible to solve _Ax_ = _b_ by the factorization _A_ = _LR_ , where _L_ and _R_ have the forms

This factorization is not always possible; but, when it is possible, it yields a faster solution of _Ax_ = _b_ than the more general method of the preceding section. The solution is faster because there are no row-interchanges. We shall give a count of operations in Section 7.4.

**E XAMPLE 1.** The factorization

is impossible because (2) implies _r_ 11 = 0. Hence, det _R_ = 0. But we must have −1 = det _A_ = (det _L_ )(det _R_ ) = _r_ 11 _r_ 22.

If _A_ = _LR,_ where all diagonal elements _r ii_ ≠ 0, the system _Ax_ = _b_ is readily solvable. Since _L_ is triangular with nonzero diagonal elements, we may solve _L_ ( _Rx_ ) = _b_ for _c_ = _Rx_ by recursion. Then _Rx_ = _c_ is solved by recursion for _x._

**E XAMPLE 2.** Let A have the factorization

To solve _Ax_ = _b_ = col (, ) we first solve

We find, successively, _c_ 1 = 1,  . We now solve

finding, successively, _x 2_ = 2, _x 1_ = 1.

**Theorem 1.** _Let A be an_ n × n _matrix. Then_ A _has a factorization_ A = LR _where_ L _is left-triangular with_ 1's _on the diagonal, and where_ R _is right-triangular with nonzero diagonal elements, if and only if_

_Moreover, the factorization_ A = LR, _if it is possible, is unique._

In particular, if _A_ is positive definite, there is a factorization _A_ = _LR,_ since all the determinants in (6) are positive, as we showed in Section 6.5. If _A_ is not Hermitian, the proof will show that the components of _L_ and of _R_ are computed in this order: row 1 of _R_ , column 1 of _L_ ; row 2 of _R_ , column 2 of _L_ ; etc. There is a simplification if _A_ is Hermitian.

_Proof of the Theorem._ First we prove the uniqueness. Let _r (i)_ denote the _i_ th row of _R_ , and _l j_ denote the _j_ th column of _L_ , with similar conventions for the rows and columns of _A_. If _A_ = _LR_ , where _L_ and _R_ have the forms (1), then the rows of the product are

The columns of _A_ = _LR_ , excepting the last, are

Since all _r jj_ ≠ 0 in (1), we may solve (7) and (8) recursively:

and so on, until at last we solve for _r_ ( _n_ −1), _l_ _n_ −1, and _r_ ( _n_ ). The last column of _L_ is known = col (0, . . ., 0, 1).

Explicitly, _r_ ( _i_ ) is given by (7) as

whereas _l i_ is given by (8) as

 .

Here we use the summation convention

Now we will show that a factorization _A = LR,_ with _L_ and _R_ of the forms (1), is possible _only if_ the subdeterminants (6) are nonzero. We observe that _A_ = _LR_ implies

Taking determinants, we find

Since (1) assumes all _r ii_ ≠ 0, we must have all subdeterminants (6) nonzero.

Finally, we will show that a factorization _A_ = _LR_ is possible _if_ the inequalities (6) hold. It is sufficient to show that there exist numbers   and _l ij_ ( _i_ > _j_ ) satisfying (10) and (11). We satisfy (10) for _i_ = 1 bydefining _r ij_ = _a ij_ ( _j_ = 1, . . . , _n_ ). Since _r 11_ = _a_ 11 ≠ 0, we may define _l i1_ from (11) with _j_ = 1. By induction, suppose that we have satisfied (10) for _i_ < _m_ and (11) for _j_ < _m_ , Where  . We now satisfy (10) for _i_ = _m_ by defining

To show that _r mm_ ≠ 0, we use (10) for _i_ = 1, . . . , _m_ and (11) for _j_ = 1, . . . , _m_ – 1 to write the matrix identity

Taking determinants, we find

Since the determinant on the right is nonzero by assumption (6), we conclude that _r mm_ ≠ 0. Therefore, (11) may be satisfied for _j = m,_ and the inductive proof is complete.

**E XAMPLE 3.** To obtain the factorization which was given in (3), write

Equations (10) for _i =_ 1 are

Equation (11) for _j_ = 1 is: _l_ 21 = 3−1(−1). Finally, equation (10) for _i_ = 2 is:  . Note that

SIMPLIFICATION FOR HERMITIAN MATRICES

If _A_ is Hermitian, there is a simple relation between _L_ and _R._ Let _A_ = _A*,_ and suppose that the subdeterminants (6) are nonzero. Then _A_ = _LR_ implies _A_ = _A*_ = _R*L*._ But _R*_ is left-triangular, and _L*_ is right-triangular. By the uniqueness of the factorization _A_ = _LR,_ we could conclude that _R_ * = _L_ and _L_ * = _R_ if the diagonal elements _r ii_ were all l's. If some _r ii_ ≠ 1, define _D_ = diag ( _r_ 11, . . . , _r nn_) Then _R_ = _DR_ 1, where  . Hence, all  . We now have

Since _R_ l has l's on the diagonal, we conclude by uniqueness:

Further, _D_ * = _D_ , since   are the diagonal elements of _D_ * _L_ *, whereas _r_ 11, . . . are the diagonal elements of _R_.

In practice, we use (19) to simplify the computation of _L_. We have _L_ = ( _R_ 1)* = ( _D_ −1 _R_ )* = _R_ * _D_ −1, or

For Hermitian _A_ , this formula replaces the longer formula (11). In (20) the _j_ th column of _L_ is given directly from the _j_ th row of _R_.

For certain applications in mathematical statistics it is desirable to construct a factorization

of a _positive definite_ Hermitian matrix _A,_ where _T_ is left-triangular. To do this, we first factor _A = LR,_ as above. The elements _r ii_, being the quotients of principal minor determinants of _A,_ are positive. Since _A_ is Hermitian, _L_ = _R_ * _D_ −1 as in (20). Now we have _A_ = ( _R_ * _D_ −1) _R_. But

Therefore, _A_ = _TT_ *, where _T_ = _R_ * _P_ , i.e.,

In particular,  .

The components _t ij_ can be computed directly. From the identity _A_ = _TT_ *, we find

Therefore, we may compute the _t ij_ in the order _t_ 11, . . . , _t_ _n_ 1; _t_ 22, . . . , _t_ _n_ 2 ; . . . ; _t nn_ from the recursion formulas

#### **PROBLEMS**

**1.** Find the components _l ij_, _r ij_ in the factorization

**2.** Find thecomponents _r ij_ in the factorization

Verify the identity   with _i_ > _j_.

**3.** Find the components _t ij_ in the factorization

Require all _t jj_ > 0.

**4.** * Prove that, if _A_ is only positive _semi_ definite, the factorization _A_ = _TT_ * is still possible, with _t ij_ = 0 for _j_ > _i_ , and  . [This result can be proved algebraically. It can also be proved analytically: Replace _A_ by _A_ \+  _I_ with   > 0. Now _A_ \+  _I_ = _T_ ( ) _T_ *( ). Why? But Σ Σ| _t jk_( )|2 = Σ( _a jj_ \+  ). Why? Let   → 0 and pick a convergent subsequence of the bounded matrices _T_ ( ).]

### **7.4 DIRECT SOLUTION OF LARGE SYSTEMS OF LINEAR EQUATIONS**

In structural engineering, in numerical analysis, and in many other fields, we encounter large systems of equations _Ax_ = _b_ in which _A_ is a band-matrix, i.e., a matrix whose nonzero components are near the main diagonal. Let _A_ = ( _a ij_) be an _n_ х _n_ matrix. _We say that A has bandwidth_ w _if_ aij = 0 _whenever _. Thus, a diagonal matrix has bandwidth _w_ = 1; a full matrix has _w = n;_ tridiagonal matrix has _w_ = 2.

**E XAMPLE 1.** Let a rectangular structure be given with base > height. For purposes of analysis we place a grid of 40 points on this rectangle, as in Figure 7.1. If the nodes are numbered in some way from 1 to 40, a symmetric matrix _A_ = ( _a ij_) is defined, from engineering considerations, in which _a ij_ ≠ 0 whenever node _i_ and node _j_ are adjacent.

To number the nodes as in Figure 7.2

**Figure 7.1**

**Figure 7.2**

produces a matrix _A_ with bandwidth _w_ = 5, since nodes _i_ and _j_ are neighbors only if | _i_ – _j_ | < 5. For example, node _i_ = 6 has the neighbors _j_ = 2, 5, 10, and 7.

If the grid in Figure 7.1 is numbered by rows, as in Figure 7.3,

**Figure 7.3**

the bandwidth _w_ = 11 results. For example, node _i_ = 12 has the neighbors _j_ = 11,2,13,22, for which | _i_ − _j_ | < 11.

**E XAMPLE 2.** Consider the ring structure in Figure 7.4(a). The nodes are to be numbered from 1 to 6. A matrix _A_ = ( _a ij_) will be defined, from physical considerations, with _a tj_ ≠ 0 when nodes _i_ and _j_ are neighbors. The natural ordering in Figure 7.4(b) gives a 6 X 6 matrix _A_ with bandwidth _w_ = 6, since nodes 1 and 6 are neighbors. However, the ordering in Figure 7.4(c) reduces the bandwidth to _w_ = 3.

**Figure 7.4**

In this section we shall discuss the factorization _A_ = _LR_ , where _L_ and _R_ are triangular as in the preceding section, and where _A_ is an _n_ × _n_ matrix of bandwidth _w._ We shall suppose

According to the last section, these inequalities guarantee the existence and uniqueness of the matrices _L_ and _R._

Formulas (10), (11), and (20) of the last section are:

The last formula replaces the one above it if _A_ is Hermitian.

We now assert that _the triangular matrices_ R _and_ L _have bandwidth_ w:

Since _r_ 1 _j_ = _a_ 1 _j_ and _l_ _i_ 1 = _a_ _i_ 1/ _r_ 11, the assertion is evident for the first row of _R_ and for the first column of _L_. For the other rows of _R_ and for the other columns of _L_ , the assertion follows by induction from (2) and (3).

**E XAMPLE 3.** In the following factorization _A_ is positive definite, _n_ = 4, and _w_ = 2.

_If_ A _is an_ n × n _band-matrix of width_ w < n, _we can shorten the calculations_ (2), (3), (4) _for_ R _and_ L. In formula (2),   needs to be computed only for   or  , whichever is smaller. In the summation (2) we have  , and the number _r kj_ vanishes unless  . Therefore, (2) becomes

In particular, as we will show,

If _i_ = 1, the upper limit of the summation (') is _i_ − 1 = 0. If _j_ = _i_ \+ _w_ − 1, the lower limit, max (1, _j_ − _w_ \+ 1), equals _i_ , which is greater than the upper limit, _i_ − 1. In either case the sum ∑ equals zero.

In formula (3), _l_ _ij_ ( _i_ > _j_ ) needs to be computed only for   or  , whichever is smaller. In the summation (3) we have _k_ < _j_ < _i_ , and the number _l_ _ik_ vanishes unless  . Therefore, (3) and (4) become

In particular,

for then the lower limit of the summation (') is _j_ > the upper limit, _j_ − 1.

COUNT OF OPERATIONS

It is not a trivial problem to count the number of operations used in the factorization _A_ = _LR_ of a band-matrix _A_. The difficulty lies in the cumbersome forms of the limits in the summations (′) and (′). We shall count the total number _μ_ of multiplications and divisions.

We shall find, for an _n_ × _n_ non-Hermitian matrix _A_ of bandwidth _w_ ,

If _A_ is Hermitian, _μ_ is reduced to

The number of additions and subtractions is substantially the same.

To calculate _μ_ we shall use a trick of combinatorial analysis. Let _L_ 1 and _R_ 1 be the special matrices denned as follows:

For example, if _n_ = 4 and _w_ = 2,

Let _A_ 1 = _L_ 1 _R_ 1. This special matrix _A_ 1 will count the number _μ_ for us. Applied to the matrix _A_ 1, formula (′) becomes

The summation in (11) counts 1 for each multiplication used in the general calculation (′) of _r ij_. Therefore, by (11), the number of multiplications in the calculation of _r ij_ equals  . For example, if _n_ = 4 and _w_ = 2,

The circled 2 in (12) shows that the general calculation of _r_ 33 requires   multiplication. Indeed, this is correct because the limits in the summation (′) for _r_ 33 are _k_ = max (l, _j_ − _w_ \+ 1) = max (1, 3 − 2 + 1) = 2, and _k_ = _i_ − 1 = 3 − 1 = 2, so that the summation does contain exactly one multiplication _l ikrkj_.

Observe that _A_ 1 is an _n_ × _n_ matrix of bandwidth _w_. Therefore, the total number of multiplicationsused to form _R_ equals

where _τ_ is the number of calculated _r ij_. Equivalently, _τ_ is the number of l's in _R_ 1. The example (10) illustrates that

Thus,

The matrix _A_ 1 is symmetric because _L_ 1is the transpose of _R_ 1. Therefore,

From this identity we can compute the sum (13). First we observe, from the product form _A_ 1 = _L_ 1 _R_ 1,

Therefore, by (15),

Let _u_ = col (1, 1, . . . , 1). Then we have the quadratic form

But

Now

Therefore, the square of the length equals

or

From (16), (17), and (21) we compute

If _A_ is non-Hermitian, formula (′) is used to compute _L_. Applied to _L_ 1, _R_ l, and _A_ l, (′) gives

The summation in (23) counts 1 for each multiplication used in (′); therefore, the number of multiplications is  . Formula (′) also contains a division by _r jj_. Therefore, _a ij_ counts the multiplications and the division used to compute _l ij_. Since   when  , the total number of multiplications and divisions used to compute _L_ is ∑   summed for _i_ > _j_. Since _A_ 1 is symmetric,

When the contribution (24) from _L_ is added to the contribution (13) from _R_ , we find, by (22) and (15),

Algebraic simplification gives the required formula (7).

If _A_ is Hermitian, the calculation (′) of _l ij_ just consists of one division. The contribution to _μ_ from _L_ is then just

Adding this to the contribution (13) from _R_ , we find

The identities (22) and (15) now yield

which reduces algebraically to (8).

To solve _LRx_ = _b_ requires some additional number _μ′_ of multiplications and divisions. Solving _Lc_ = _b_ for _c_ requires

multiplications. Solving _Rx_ = _c_ for _x_ requires

multiplications and divisions. The sum of (27) and (28) is

The total number of multiplications required to solve _Ax_ = _b_ is _μ_ \+ _μ′_.

#### **PROBLEMS**

**1.** Consider electrical network of resistors and one battery in Figure 7.5.

**Figure 7.5**

If there is a resistor between nodes _i_ and _j_ , call the resistance _R ij_ = _R ji_. Suppose that equations were written for the unknown voltages _E_ 1, . . . , _E_ 100. For example, the equation at node 5 is

The equation at node 2 is

Suppose that the equations in matrix-vector form are _Ax_ = _b_ , where _x_ = col ( _E_ 1, . . . , _E_ 100). The vector _b_ is nonzero; what are its components? The 100 × 100 matrix _A_ is a band-matrix; what is its bandwidth, _w_? What are the components in the 84th row of _A_?

**2.** Show that the matrix _A_ in Problem 1 is symmetric. Show that the quadratic form of _A_ equals

whereif   if there is no resistor between nodes _i_ and _j._ Conclude that _A_ is positive definite. (HINT: The quadratic form of _A_ is

Reverse the indices _i_ and _j._ Add the two expressions for ( _Ax_ , _x_ ), and divide by 2. Use the symmetry of _R ij_).

**3.** Why can the matrix _A_ in Problem 1 be factored in the form _A_ = _LR_? How many multiplications, _μ_ , are needed for this factorization? What is the total number of multiplications, _μ_ \+ _μ_ ′, needed to solve for the unknown potentials _E_ 1, . . . , _E_ 100?

**4.** A full 100 × 100 matrix requires 10,000 computer-storage locations. How many computer-storage locations are required to store the symmetric band-matrix _A_ in Problem 1?

**5.** Suppose that the voltages in Problem 1 were renamed as follows: On the upper row, call the unknown voltages _V_ 1, _V_ 2, . . . , _V_ 50; on the lower row, call the unknown voltages _V_ 51, _v_ 52, . . . , _V_ 100. If a matrix-vector equation is written for the renamed voltages, what is the bandwidth of the matrix?

**6.** Define the _mesh_ currents _i_ 1, _i_ 2, . . . , _i_ 50 as indicated in Figure 7.5. The current between nodes 1 and 3 is _i_ 2. The net current between nodes 5 and 6 is the difference _i_ 3 − _i_ 4. The equations for the mesh currents are

etc. If these equations are written in matrix-vector form, _Ax_ = _b_ , show that _A_ is a 50 × 50, _tridiagonal_ ( _w_ = 2) matrix.

**7.** Show that the matrix _A_ in Problem 6 is symmetric and positive definite.

**8.** How many multiplications, _μ_ \+ _μ_ ′, are needed to solve for the mesh currents in Problem 6? Problems 1, 5, and 6 present three different mathematical formulations of the same physical problem. Which of these formulations is the best for computation?

**9.** _The continued fractions related to a triadiagonal matrix:_ Let

and assume

Show that

where the unknowns _r_ 1, . . . , _r_ 4 are given by

Replace 4 by _n_ and generalize this result.

**10.** In the factorization discussed in Problem 9, if _A_ is Hermitian and positive definite, show that

**11.** * In Problem 10, show that all   the smallest eigenvalue of _A_. Use the inclusion theorem, [Section 6.4, and the identity _r_ 1 . . . _r_ v = det (a _ij_ ) ( _i_ , _j_ = 1, . . . , _v_ ), Section 7.3.]

### **7.5 REDUCTION OF ROUNDING ERROR**

Digital computers do not do arithmetic exactly. A number stored in a digital computer has only a certain number of significant figures. For example, the number may be stored in binary, floating-point form as

This number has an exponent _e_ = 5, and it has _s_ = 8 significant figures, called "bits" in the binary form. The numerical value of _x_ is

If only _s_ significant figures are conserved, computation produces rounding error. For example, if

The number _x_ \+ _y_ is computed as follows:

If _z_ is the computed value of the sum in single precision, with _s_ = 8, we write

The number equal to _z_ results from rounding the precise sum (4) to 8 significant bits; in this case we round up, since the first neglected bit is a 1.

We shall use the notation   ... to mean that _x_ is the computed value of... if the computation is performed (by some prescribed algorithm) in single precision. If the computation is performed in double precision, we shall write   . . . . If _s_ significant bits are used in single precision, then 2 _s_ bits are used in double precision; in current practice, _s_ = 27 or some greater number. The sign "=" shall designate exact equality.

**E XAMPLE 1.** Let _s_ = 3. Let

Let

Since _y_ = 26(.0000111), we have

Observe that the true sum _u_ is rounded down to produce _v,_ since the first neglected bit is 0; _u_ is rounded up to produce _w,_ since the first neglected bit is 1.

Rounding error in matrix computations is discussed at length in a book by J. H. Wilkinson. Here we shall only present two simple suggestions and give one crude analysis. Our arguments will be not mathematically rigorous, but only suggestive.

The first suggestion is to _accumulate inner products in double precision_ before rounding to single precision. Whenever an expression _x_ 1 _y_ 1 \+ ... + _x_ k _y_ k has to be computed, there is a great gain in accuracy if

are successively computed in double precision. The final value is then rounded to _s_ significant bits, and the computation proceeds. Of course, if there are time and memory enough to perform the whole computation in double precision, so much the better.

The second suggestion is more elaborate. Suppose that we have solved a large system of equations _Ax_ = _b_ in single precision by some algorithm, say that of the preceding section. If the computed solution is called _c_ , we write

To assess the error, we compute the residual vector _b_ − _Ac_ in double precision, obtaining

We now try to diminish the residual _b_ − _Ac_ by computing

and by forming the new vector

_The vector_ y _is usually an improved value for_ A-1b; in other words, we assert that usually

if | | is some convenient norm (see Section 6.9). The assertion (12) is motivated as follows: If _e_ were the exact value of the residual, _b_ − _Ac_ , and if _d_ were exactly _A_ − _e_ , then we should have

The objection, of course, is that we have used a single-precision solution (10) to correct a single-precision solution (8). This objection is usually not valid because _c_ equals _A_ −1 _b_ apart from an error which is usually a small fraction of _b_ in norm. The vector e practically equals _b_ − _Ac_ because _e_ is computed in double precision. From (10), the vector _d_ equals _A_ −1 _e_ apart from an error which is a small fraction of e, and therefore a _very_ small fraction of _b._ Thus, _d_ has less absolute error than c does, and it makes sense to add _d_ as a correction to _c._

Now we will make the reasoning in the preceding paragraph more precise. Refer to the computational equations (8)–(11). We shall estimate the errors c − _A_ −1 _b_ and _y_ − _A_ −1 _b_. The error _c_ − _A_ −1 _b_ can often be estimated by some inequality

where | _A_ −1| is the norm of _A_ −1, | _b_ | is the norm of _b_ , _s_ is the number of significant bits in single-precision computation, and _α_ is some not-too-large constant depending only on the prescribed solution algorithm and on the number _n_ of components in the vector _b_. Similarly, for the error in the double-precision computation (9), we write

where _β_ is some constant like _α_. Following (13), we write for the error in the single-precision solution (10)

Finally, from (11) we write

where _γ_ is some constant like _α_ and _β_.

We can estimate _y_ − _A_ −1 _b_ from the preceding inequalities.

Therefore,

The quantities _b_ and _A_ are given; we must estimate | _c_ |, | _d_ |, and | _e_ |. From (13),

From (14), (13), and (19) we find

Now (15) gives

As just defined, the numbers _ρ_ c, _ρ_ e, _ρ_ d are of moderate size unless | _b_ |, | _A_ |, or | _A_ −1 | is large. Regarding _s_ as a parameter, we may say that _c_ is of order 1, while _e_ and _d_ are of order 2− _s_. From (18) we now find

Thus, as a function of _s_ , the original error _c_ − _A_ −l _b_ is of order 2− _s_ ; but the new error _y_ − _A_ −l _b_ is of order 2−2 _s_. For further analysis, one could put the right-hand side of (22) in an explicit form by evaluating the constants ρc ρ _e_ and ρ _d_ from the definitions (19), (20), (21).

#### **PROBLEMS**

**1.** Let _x_ 1 and _x_ 2 be two binary numbers

Here we have _s_ = 4 significant bits. Assume  , and assume _a_ 1 = _b_ 1 = 1. Form the single precisionsum or difference

Let there be rounding up or rounding down, depending upon whether the first neglected bit is a 1 or a 0. Let _z_ = _x_ 1 ± _x_ 2. Show that

Deduce that _y_ can be written in the form

where  and  .

**2.** Let _x_ 1 and _x_ 2 be defined as in Problem 1. Let

Show that   where | | < 2− _s_.

### **7.6 THE GAUSS-SEIDEL AND OTHER ITERATIVE METHODS**

To solve large systems of equations _Ax_ = _b_ , particularly those occurring in the numerical analysis of partial differential equations, we may use an iterative method. Beginning with an initial guess _x_ 0, we compute an improved guess _x_ 1. From _x_ l we obtain an improved guess _x_ 2, etc. The method converges if, for every initial guess _x_ 0, the vectors _x_ 1, _x_ 2. . . tend to the true solution _x_.

The simplest iterative method is the Gauss-Seidel method. For notational simplicity we shall write _u_ = _x k_ and _v_ = _x_ _k_ +1; the manner of forming _x_ _k_ +1 from _x k_ is independent of _k_. The first guess _x_ 0 is chosen in any convenient manner, e.g., _x_ 0 = 0 or _x_ 0 = some vector which appears to approximate the desired solution _x_.

If all _a_ _ii_ ≠ 0, for _n_ = 3 the system _Ax_ = _b_ may be written in the form

The equations (1) suggest the following scheme for obtaining an improved guess _v_ from a given guess _u_ :

Observe that, after a component _υ i_ has been computed, it is used in all subsequent equations. Thus, _υ_ 1 appears on the right-hand sides of the last two equations; _υ_ 2 appears on the right-hand side of the last equation. For general _n_ the scheme is

Thus, for _k_ = 0, 1, 2, ... we define an infinite sequence of vectors _x_ 1, _x_ 2, . . . by the scheme

We assume det _A_ ≠ 0 and all _a ii_ ≠ 0. We ask whether

If the _k_ th error-vector is _d k_ = _x k_ − _x_ , the limit (5) exists if _d k_ → 0 as _k_ → ∞. For notational simplicity, let

Subtracting the equations

from the equations (3), we find

Define the matrices _L_ and _R_ as follows:

Thus, _A_ = _L_ \+ _R_. By (7),

In other words, _Le_ = − _Rd_ , or

Applying (10) repeatedly, we find

On the left-hand side of (11), _k_ is a superscript; on the right-hand side, _k_ is an exponent. We now show:

**Theorem 1.** _Let_ A _be an n_ × _n matrix with_ det A ≠ 0 and all a11 ≠ 0. _Then the vectors_ x1, x2, . . . _defined by the Gauss-Seidel method_ (4) _converge to the limit_ x = A−1 b _for every initial guess_ x0 _if and only if all roots_ λ _of the equation_

_lie in the unit circle_ |λ| < 1.

**E XAMPLE 1.** For _n_ = 2 the equation (12) becomes

The roots of this equation are _λ_ = 0 and _λ_ = _a_ 12 _a_ 21/( _a_ 11 _a_ 22). Thus, for _n_ = 2, the Gauss-Seidel method converges for every initial guess if and only if | _a_ l2 _a_ 2l| < | _a_ 11 _a_ 22|.

_Proof of_ _Theorem 1_. Equation (11) states

The vector _x_ 0 − _x_ is arbitrary because _x_ 0 is arbitrary. Letting _M_ = − _L_ −1 _R_ , we ask when the vectors _M_ _k_ _d_ 0 tend to zero as _k_ → ∞ for every vector _d_ 0. Clearly, the vectors _M kd_0 tend to zero if the matrices _M k_ tend to the zero matrix. If  , then some component, say the _r_ , _s_ component, of _M k_ does not tend to zero. Then if _d_ 0 equals the unit vector _e s_, the _r_ th component of the vectors _M kes_ cannot tend to zero. _Hence_ , xk x→ x for all x0 _if and only if the matrix powers_ Mk _tend to_ O.

Theorem 3 of Section 4.9 states that _M k_ → O if and only if all eigenvalues _λ_ of _M_ lie in the unit circle | _λ_ | < 1. But

Since det _L_ is the nonzero constant Π _a_ _ii_ we have det ( _λI_ − _M_ ) = 0 if and only if det ( _λL_ \+ _R_ ) = 0; but this is equation (12), and the proof is complete.

Next we shall present a general theorem of H. B. Keller which implies that _the Gauss-Seidel method converges for every positive-definite Hermitian matrix_ A.

**Theorem 2.** _Let_ A _be an_ n × n _positive-definite Hermitian matrix. Let_ N _be any nonsingular_ n × n _matrix for which the Hermitian matrix_ N + N* − A _is positive definite. Given an initial vector_ x0, _form_ x1, x2, . . . _from the iterationscheme_

_Then, for every_ x0, xk → A−1b _as_ k → ∞.

In the Gauss-Seidel method (3), we have

where the matrices _L_ and _R_ are defined in (8). In the notation of (15),

The diagonal matrix is positive definite because all _a ii_ > 0 in positive-definite matrix _A_. Thus, the hypothesis of Theorem 2 is satisfied.

_Proof ofTheorem 2._ The solution _x_ = _A_ -l _b_ satisfies _Nx_ = _b_ \+ ( _N_ − _A_ ) _x_. Subtraction from (15) yields

Therefore,

As in the proof of Theorem 1, _x k_ → _x for every initial vector_ x0 _if and only if all eigenvalues_ λ _of the matrix_ I − N−1A _lie in the unit circle_ |λ| < 1.

Let _λ_ be an eigenvalue of _I_ − _N_ -l _A_ , and let _u_ ≠ 0 be an eigenvector. We will show that |λ| < 1. From ( _I_ − _N_ −l _A_ ) _u_ = _λu_ we find, after multiplying by _N_ ,

Taking the inner product of both sides with _u_ , we find

Since the right-hand side is positive, _λ_ ≠ 1. The conjugate of equation (21) is

because  , Let _λ_ = _α_ \+ _iβ_. Then (21) and (22) yield

Because _N_ \+ _N_ * − _A_ is positive definite, the left-hand side is > ( _Au_ , _u_ ). Division of (23) by ( _Au_ , _u_ ) now yields

and hence _α_ 2 \+ _β_ 2 < 1. Thus, | _λ_ |2 < 1, and the proof is complete.

RATE OF CONVERGENCE

In any iterative scheme of the form _Nx_ _k_ +1 = _b_ \+ ( _N_ − _A_ ) _x_ _k_ the error _x_ _k_ − _x_ equals ( _I_ − _N_ −l _A_ ) _k_ times the initial error. For simplicity, suppose that the initial error has an expansion

in the eigenvectors _u j_ of the matrix _T_ = _I_ − _N_ −1 _A_. Let _T_ have the eigenvalues _λ j_, with | _λ_ 1| > | _λ j_| for _j_ > 1. Then

where

Then, if _ξ_ 1 ≠ 0,

If the norm | _u_ 1| = 1, we have the norm of the error

Thus, _the rate of convergence is determined by the dominant eigenvalue λ_ 1.

If   > 0 is very small, the number _v_ of iterations required so that   is determined approximately by the equation  . Therefore,

where

The number _v e_ is approximately equal to the number of iterations necessary to divide an initial error by the factor _e_ = 2.718 . . . , since | _λ_ 1| raised to the power _v e_ equals 1/ _e_. The number _v e_ depends only on the matrices _A_ and _N_ ; _v e is a measure of the slowness of the iteration scheme_ Nxk+1 = (N − A)xk \+ b. A different scheme, with a measure of slowness  , is twice as slow if  . _The rate of convergence may be defined as_ 1/ _v e_ = log (1/| _λ_ 1|).

The assumption (25) that the initial error _x_ 0 − _x_ can be developed in a series of eigenvectors of _T_ , is not important; nor is it important that _T_ have a single dominant eigenvalue of multiplicity one. An initial error _x_ 0 − _x_ has an expansion

in principal vectors (see Section 5.2) associated with the different eigenvalues _λ_ 1, _λ_ 2, . . . , _λ_ 3 of _T_. If _λ j_, has multiplicity _m j_, then

The series terminates with _v_ = _m j_ − 1 because the principal vector _p j_ has grade  . If _g_ = 0, then _p j_ = 0. If   then (32) implies

where _γ j_ is the constant (independent of _k_ ):

Here we have used the asymptotic form for the binomial coefficient:

Let  . Let _g_ be the largest grade of principal vectors _p j_ of maximum modulus | _λ j_| = | _λ_ 1|. Assume | _λ 1_| > 0. Then _T k_ ∑ _P j_ behaves, as _k_ → ∞, like a bounded vector times the scalar _k_ _g_ −1| _λ_ 1| _k_. Define _v e_ to be the average number of iterations needed to divide the error by the factor _e_ = 2.718 . . . . Then for large numbers _r_ of repetitions of _v e_ iterations, we should have the approximate equation

Taking logarithms, we find

Dividing by _r_ and letting _r_ → ∞, we obtain

which yields _v e_ = 1/log (1/ _λ_ 1|), as in (30).

Faster iterative methods are based on the choice of _N_ to reduce | _λ_ 1|. _Successive over-relaxation_ , in a simple form, relies on the choice of the parameter _ω_ > 1 to reduce | _λ_ 1| for the scheme

where _u_ = _x k_ and _v_ = _x_ _k_ +1. Such methods are studied in _Matrix Iterative Analysis_ by R. S. Varga (Prentice-Hall, Inc., 1962).

#### **PROBLEMS**

**1.** Let _A_ be an _n_ × _n_ Hermitian matrix, and assume all _a ii_ > 0. Let _e k_ be the _k_ th. unit vector. If a vector _u_ is given, show (by calculus) that the unique value of the real parameter _λ_ which minimizes [ _A_ ( _u_ − _λe k_),( _u_ − _λe k_)] is

**2.** Let _A_ be an _n_ × _n_ , nonsingular Hermitian matrix with all _a ii_ > 0. Let _x_ and _y_ be successive iterates in the Gauss-Seidel process for solving _Az_ = _b_. Let _u_ and _v_ be the successive errors _u_ = _x_ − _z_ , _v_ = _y_ − _z_. From the result of Problem 1, show that

with equality if _v_ = _u_ = 0. (Note that _v_ is formed from _u_ by a succession of _n_ transformations of the form _u_ → _u_ − _λe k_. Note also that _v_ = _u_ implies _Au_ = 0, which implies _u_ = 0.)

**3.** Let _A_ be an _n_ × _n_ nonsingular Hermitian matrix with all _a ii_ > 0. From the result of Problem 2, prove that the Gauss-Seidel method converges for the matrix _A only_ if _A_ is positive definite.

**4.** Generalize Theorem 1 for the successive over-relaxation scheme (37).

**5.** * Let

For which real values of _ω_ does the successive over-relaxation method (37) converge? For which value of _ω_ is the rate of convergence maximized? Compare the maximum rate of convergence with the rate for the Gauss-Seidel method ( _ω_ = 1). Graph the rate of convergence as a function of _ω_.

### **7.7 COMPUTATION OF EIGENVECTORS FROM KNOWN EIGENVALUES**

Later we will discuss the computation of eigenvalues, but we hardly need to discuss the computation of eigenvectors. _If a simple eigenvalue is known, a corresponding eigenvector can be computed by solving a linear, inhomogeneous system of equations._

**E XAMPLE 1.** Suppose that, for the matrix

we have somehow computed an eigenvalue _λ_ = 1. We wish to compute a corresponding eigenvector _x_ = col ( _x_ 1, _x_ 2, _x_ 3). At least one of the components _x i_ must be nonzero. We assume, say, _x_ 3 ≠ 0. If this highly probable assumption is correct, we may normalize the eigenvector _x_ by requiring _x_ 3 = − 1. The equation ( _A_ − _λI_ ) _x_ = 0, with _λ_ = 1, now takes the form

Here we have three equations in two unknowns, _x_ 1 and _x_ 2. But _the last equation is redundant_ , being the second equation minus the first. Therefore, we need only to satisfy the first two equations

This system has the unique solution  , _x_ 1 = 0. Thus, we have computed the eigenvector   belonging to the eigenvalue _λ_ = 1.

In general, let _A_ be an _n_ × _n_ matrix for which we have computed an eigenvalue. We wish to compute a corresponding eigenvector _x_ ≠ 0 satisfying the singular system

**Theorem 1.** _Let λ be a_ simple _eigenvalue of an_ n × n _matrix_ A. _Let_ Br _be the_ (n − 1) × (n − 1) _matrix formed by omitting row_ r _and column_ r _from_ B = A − λI. _Then at least one of the matrices_ Bl, . . . , Bn _is nonsingular._

_Proof._ Let _μ_ be a scalar variable. Let   Since _λ_ is a simple eigenvalue,

But, as we saw in Section1.8,   equals

Therefore, at least one det _B r_ ≠ 0.

This result implies that the matrix _A_ − _λI_ has rank _n_ − 1, and therefore that _all eigenvectors_ x _belonging to the simple eigenvalue λ are scalar multiples_ x = _α_ x0 of any one eigenvector x0.

**Theorem 2.** _Let λ be a simple eigenvalue of the_ n × n _matrix_ A. _In the notation of the last theorem, let_ det Bq ≠ 0. _Then an eigenvector_ x _can be computed by setting_ xq = 1 _and by solving the nonsingular system_

_Proof_. The system (7) is nonsingular because its matrix is _B q_, and det _B q_ ≠ 0. We will now show that the unique solution of (7) solves the extra equation

Consider the singular matrix

If we add _x j_ times column _j_ to column _q_ for all _j_ ≠ _q_ , the result is a vector, _υ_ , with components _υ i_ = 0 for all _i_ ≠ _q_ that is what (7) says. Expanding the determinant of the resulting matrix by column _q_ , we find

where

Since det _B q_ ≠ 0, (10) implies _υ q_ = 0; that is the extra equation (8). If we now define

we have ( _A_ − _λI_ ) _x_ = 0, _x_ ≠ 0, as required.

If _λ_ is a simple eigenvalue, the practical application of Theorem 2 requires only a little luck. We must choose _q_ so that the _B q_ is nonsingular. By Theorem 1, _some_ value _q_ will succeed. Usually det _B q_ ≠ 0 for all _q_. In practice, we first try _q_ = _n_ , as in Example 1. If _q_ = _n_ fails, we try some _q_ ≠ _n_.

If _λ_ is a multiple eigenvalue, the problem is more difficult. If _A_ − _λI_ has rank _r_ < _n_ , there are _v_ = _n_ − _r_ independent eigenvectors _x_ 1, . . . , _x v_ corresponding to _λ_. In principle, the eigenvectors _x k_ can be found by first locating _r_ independent columns of _A_ − _λI_. Let the _other_ columns of _A_ − _λI_ be columns _j_ 1, . . . , _j v_. Set

Now the equation ( _A_ − _λI_ ) _x_ = 0 can be rewritten as a nonsingular system of _r_ equations in the _r_ unknowns  .

In practice, the rank _r_ of _A_ − _λI_ is usually unknown. If _λ_ has multiplicity _m_ , we can show only that  . Further, the rank can seldom be computed numerically, since rounding error will produce nonzero values for certain determinants whose exact value is zero.

#### **PROBLEMS**

**1.** Let _A_ be an _n_ × _n_ matrix, and let _λ_ be an eigenvalue. If the rank of _A_ − _λI_ is _r_ , why are there _n_ − _r_ independent eigenvectors corresponding to _λ_? Can there be more than _n_ − _r_ independent eigenvectors? (Use Theorem 5, Section 2.4).

**2.** For the matrix

what is the multiplicity of the eigenvalue 7? What is the rank of _A −_ 7 _I_? What is the greatest number _v_ such that _A_ has _v_ independent eigenvectors corresponding to the eigenvalue _λ_ = 7? Calculate _v_ independent eigenvectors.

**3.** Calculate an eigenvector for the matrix

corresponding to theeigenvalue _λ_ = −2.

**4.** Calculate two linearly independent eigenvectors for the matrix

corresponding to the eigenvalue _λ_ = −2. Use formula (13) with _j_ 1= 3 and _j_ 2 = 1. Theeigenvectors will have the forms

**5.** Let _λ_ be an eigenvalue of multiplicity _m_ for the _n_ × _n_ matrix _A_. Prove that

(Use the method used to prove Theorem 1; here we have

**6.** In Problem 5, show that all the ranks _r_ satisfying   can actually occur for certain matrices _A_. Illustrate all cases _r_ = 1, 2, 3 when _n_ = 4 and _m_ = 3.

### **7.8 NUMERICAL INSTABILITY OF THE JORDAN CANONICAL FORM**

In the preceding section we had an example of _numerical instability_. The problem was to compute the rank _r_ of a singular matrix _B_ = _A_ − _λI_. The integer _r_ depends unstably on the data _B_. This is because an arbitrarily small perturbation of _B_ produces a matrix _B_ ′ whose rank _r_ ′ may differ from _r_ by a nonzero integer. Even if the data _B_ = ( _b ij_) are stored with perfect precision, the method of computing rank by looking for the largest nonzero subdeterminant, is numerically unstable because an arbitrarily small error in the computation produces nonzero values for determinants whose true value is zero.

In general, if we wish to compute a quantity _y_ from data _x_ by some numerical method _M_ , we may say that _y_ depends unstably on _x_ if small errors in the data _x_ produce large errors in the computed quantity _y_ even if the method _M_ is executed with no rounding error. The method _M_ is called numerically unstable if small rounding errors in the execution of _M_ produce large errors in the computed value _y_ even when the data are exact. _As a rule, one should not try to compute numerically unstable quantities, and one should not use numerically unstable methods_.

If _A_ is an _n_ × _n_ matrix with multiple eigenvalues, quantities of great interest are the Jordan canonical form _J_ and the associated matrix _B_ of principal vectors (see Chapter 5). Unless _A_ is known to be Hermitian or to have some other useful structure, _the quantities_ J _and_ B _depend unstably on_ A.

**E XAMPLE 1.** If   ≠ 0, and if _T_ is nonsingular, the matrix

has the Jordan form

But   = 0 yields the form _J_ = diag (1, 1).

**E XAMPLE 2.** If   = 0, the matrix

has the Jordan form  , where

But if   ≠ 0, we have  , _where_

The matrices _J _ and _B _ do not approach _J_ 0 and _B_ 0 as   → 0.

Sometimes a numerical method _M_ is unstable even though the quantity to be computed, _y_ , depends stably on the data, _x_. But if _y_ depends unstably on _x_ , every numerical method _M_ will, as a rule, be unstable. The reason is that the method _M_ prescribes certain computations: _u_ = _f_ ( _x_ ), _υ_ = _g_ ( _u_ , _x_ ), . . . , _y_ = _h_ ( _x_ , _u_ , _υ_. . .). At the first stage, suppose that rounding error produces an inexact value _u′_. This value _u′_ is exactly equal to _f_ ( _x′_ ), where _x′_ is not exactly equal to _x_. Even if all later computations were exact, the computed value _y′_ would be the exact value resulting from data _x′_ ≠ _x_. If the computed value depends unstably on the data, we must expect _y′_ to be far from the true value _y_.

For this reason, all methods for computing the Jordan form are numerically unstable.

#### **PROBLEMS**

**1.** * Instead of saying that the quantity _y_ depends _unstably_ on the quantity _x_ , why don't we use the familiar, precise mathematical term: _discontinuously_? (Do numbers _x_ in a digital computer range continuously in intervals _a_ < _x_ < _b_?)

**2.** * Suppose that we wish to compute _y_ = 3.1416 _x_. The quantity _y_ depends stably on the data _x_. Let _M_ be the numerical method of computing _y_ from _x_ by the formula

Suppose that we are computing in single precision with 32 binary digits (bits) per word. Is the method _M_ numerically stable?

**3.** Let a scalar _y_ be computed from scalars _x l_. . . , _x n_. by the formula _y_ = _f_ ( _x_ 1, . . . , _x n_). Suppose that the computation is executed with a small rounding error  . If _f_ is differentiate, give an approximate condition for data errors  1, . . . ,  n to produce the same final error,  . In other words, when do we have

**4.** * Suppose that, for all   in the range 0 < |   | <  0, the matrices _A_ ( ) have a complete set of _n_ independent unit eigenvectors _u_ l( ), . . . , _u n_( ). Let _T_ ( ) be the square matrix whose columns are _u_ l( ), . . . , _u n_( ). Suppose that, for all positive |   | <  0,

where _δ_ is independent of  . Suppose that _A_ ( ) → _A_ as   → 0. Prove that the limit, _A,_ has a complete set of _n_ independent unit eigenvectors _u_ 1, . . . , _u n_. Conclude that _A_ has a canonical diagonalization _T_ −1 _AT_ = Λ.

**5.** * Let _u_ 1, . . ., _u n_ be unit vectors in _E n_. We define a measure _μ_ of independence by the formula

for all coefficients _c_ l . . . , _c n_ normalized by the condition Σ | _c j_ |2 = 1. By || || we mean the Euclidean norm. Let _T_ be the matrix whose columns are _u_ l . . . , _u n_. Show that _μ_ 2 is the least eigenvalue of _T*T_. Conclude that

**6.** Let matrices _A_ ( ) → _A_ as   → 0. Suppose that, for |   | ≠ 0, each matrix _A_ ( ) has a complete set of independent unit eigenvectors _u_ 1( ), . . ., _u n_( ). From the results of the last two problems prove that the limit, _A_ , has a complete set of independent unit eigenvectors _u_ l, . . ., _u n unless_ the sets _u_ l( ), . . . , _u n_( ) become dependent, i.e., unless _μ_ ( ) → 0, where _μ_ is the measure of independence defined in Problem 5.

**7.** * Using Hadamard's inequality (Section 6.6) show that the measure of independence _μ_ denned in Problem 5 satisfies the inequality  , with _μ_ = 1 if and only if the unit vectors _u_ l, . . . , _u n_ are mutually orthogonal.

**8.** Let λ be a simple root of the equation

How does _λ_ vary if _c_ l, . . . , _c n_ are replaced by nearby values _c_ 1 \+  1, . . . , _c n_ \+  _n_? Does _λ_ depend stably on the data _c_ 1, . . . , _c n_ if   is small?

* * *

 J. H. Wilkinson, _Rounding Errors in Algebraic Processes._ Englewood Cliffs, N.J.: Prentice-Hall, Inc., 1963.

### **7.9 THE METHOD OF ITERATION FOR DOMINANT EIGENVALUES**

In this section we shall present a method for computing the dominant eigenvalue or eigenvalues of an _n_ × _n_ matrix _A._ To take the most useful case, we shall assume that _A_ is real. In scientific and engineering applications, we are typically interested in finding the real and the complex eigenvalues of a _real_ matrix. Our analysis could readily be extended to matrices with complex components. In Section 7.10 we will show how to compute the smaller eigenvalues.

A real matrix _A_ has a characteristic polynomial det ( _λI_ − _A_ ) with real coefficients. Therefore, the eigenvalues of _A_ occur as real numbers or as complex-conjugate pairs, _λ_ and _ ._ For clarity, we shall suppose that the eigenvalues _λ_ 1, _λ_ 2, . . . , _λ n_ of _A_ are distinct, so that _A_ has _n_ linearly independent eigenvectors _u_ 1, _u_ 2, . . . , _u_ n. (Without difficulty the reader will be able to generalize this analysis, using either the Jordan form of _A_ or an expansion in principal vectors, as in Section 7.6.)

Let  . The _dominant_ eigenvalues are defined to be the eigenvalues ot greatest absolute value. We shall consider only two cases.

_Real case._ Here we suppose that _A_ has a single real eigenvalue _λ_ 1 of maximum absolute value. We allow _λ_ 1 to be either positive or negative. Thus,

_Complex case._ Here we suppose that _A_ has the complex-conjugate dominant eigenvalues

Other cases are possible but unusual, and we ignore them. For example, _A_ could have the dominant eigenvalues +1 and −1. Or _A_ could have the five dominant eigenvalues _λ k_ = −7 exp (2 _kπi_ /5) ( _k_ = 1, . . . , 5). Observe that, in both of these examples, the translation _λ k_ → _λ k_ \+ 1, caused by adding 1 to each diagonal element of _A_ , produces eigenvalues _λ k_ \+ 1 which fall into the real case (1), or the complex case (2).

In the real case (1), we pick an initial vector _x_ 0 and form the iterates _x_ 1 = _Ax_ 0, _x_ 2 = _Ax_ 1, . . . . If _x_ 0 has the expansion

then the iterate _x k_ has the expansion

where, since   for all  .

We suppose that _ξ_ 1 ≠ 0. This supposition is almost surely correct if _x_ 0 is picked in some bizarre way, e.g.,

Then for large _k_ , as (5) shows,

For large _k_ , successive vectors are approximately multiples of each other, i.e., they are approximately eigenvectors:

In the complex case (2), the iterates _x k_ = _A kx0_ do _not_ approximate eigenvectors as _k_ → ∞.

**E XAMPLE** 1. Let

The eigenvalues are λ1 = 1 + _i_ , λ2 = 1 − _i_. The iterates _x_ 0, x1,. . . are

The vector _x k+l_ is found by rotating _x k_ through 450 and multiplying the length by  .

In general, let _x 0_ have an eigenvector expansion (3). Even though we are computing complex eigenvalues λ1 and   we will use only real numbers until the last step. We pick for _x_ 0 a _real_ vector such as the vector defined by (7). The iterate _x k_ has the eigenvector expansion (4). Let

Since   for _v_ > 2, the expansion (4) yields

where _r k_ → 0 as _k_ → ∞. Since _A_ and _x k_ are real, we may assume that   and that the eigenvectors _u 1_ and _u 2_ are complex conjugates. _We assume that ξ_ 1 ≠ 0. Note that, for all _k_ , since _u_ 1 and _u_ 2 are independent,

where  . Therefore, we have the asymptotic form

In the complex case for large _k_ , successive vectors _x k_ do not become multiples of each other. For large _k_ , the vectors _x k_ lie approximately in the plane spanned by the independent eigenvectors _u 1_ and _u 2._ Therefore, _successive triples x k, xk+1, xk+2 are almost dependent._ Explicitly, if _α_ and _β_ are the real coefficients of the quadratic

then, for all _k_ ,  , and hence

The asymptotic form (14) now yields the approximate equation

If we can find the coefficients of dependence _α_ and _β_ we can compute λ1 and λ2 as the roots of the quadratic (15):

Since λ1 and λ2 are complex, _α_ 2 – 4 _β_ < 0.

**E XAMPLE 2.** In Example 1, Since _λ_ 1 = 1 + _i_ and _λ_ 2 = 1 − _i_ are the only eigenvalues, the iterates (11) satisfy

Here _α_ = –2, _β_ = 2, ( _λ_ − _λ_ 1)( _λ_ − _λ_ 2) = _λ_ 2 − 2 _λ_ \+ 2.

_Computing procedure._ In practice, we are given an _n_ × _n_ real matrix _A_ ; we assume that we have either the real case (1) or the complex case (2), _but we do not know which_. If we have the real case, we must compute _λ 1_; if we have the complex case, we must compute λ1 and

Begin with the vector _x_ = _x_ 0 defined, say, by (7). Compute the first iterate _y_ = _Ax_. Now make a least-squares fit of _x_ to _y_ , i.e., choose the real number _λ_ to minimize

Setting the derivative with respect to λ equal to zero, we find the value

Define a small tolerance   > 0, for example,   = 10−5. If the quantities _x_ , _y_ , and _λ_ satisfy

we conclude that we have the real case, and we assign the value _λ_ to _λ_ 1.

If the test (21) is failed, we compute the third iterate, _z_ = _Ay._ Pick the real numbers _α_ and _β_ which minimize

Calculus gives the necessary conditions

The optimal values _α_ and _β_ are found from (23) to be

If the quantities _x,y, z, α, β_ satisfy

we conclude that we have the complex case, and we assign the root values (18) to _λ_ 1 and _λ_ 2.

Suppose that both the real test (21) and the complex test (25) are failed. Then we must iterate further. If we form the iterates _A kx_ without some _scaling_ , we may eventually obtain numbers too large or too small for the floating-point representation. The last computed iterate was _z = A 2x._ We scale _z_ by dividing _z_ by the component _z m_ of largest absolute value. We then define

We now repeat the preceding process: We form _y_ = _Ax_ and make the real test (21). If that test is failed, we form _z_ = _Ay_ and make the complex test (25). If the complex test is also failed, we define a new vector _x_ by (26), etc. If   is not too small, one of the tests ultimately will be passed. Scaling does not affect the asymptotic equation for _x k_ = _x_ , _x_ _k_ +1 = _y_ , and _x_ _k_ +2 = _z_ :

because the equation is homogeneous. The sequence of scaled vectors _x; y, z; x, y, z;_. . . is simply

where σ1, σ2,. . . are nonzero scalars.

_The method of iteration suffers very little from rounding error_. If a new _x_ is slightly in error, we may simply regard this _x_ as a new initial vector which is, no doubt, better than the previous initial vector. Rounding error affects the accuracy of the computed eigenvalues only in the last stage of the computation, when _x_ is fitted to _y,_ or when _x_ and _y_ are fitted to _z_.

The reader may have wondered why, if we are computing eigenvalues, we do not simply find the coefficients of the characteristic polynomial _φ_ ( _λ_ ) = det ( _λI_ − _A_ ). Then the eigenvalues _λ j_ could be found by some numerical method for solving polynomial equations   Indeed, this is sometimes done. But unless an extremely efficient method such as that of Householder and Bauer (see Section 7.12) is used, there is a large rounding error from the extensive computations needed to obtain the _n_ coefficients of   from the _n 2_ components of _A_. Even for _n_ = 5 or 6 the rounding error may be quite large.

Moreover, a root _λ j_ of   depends unstably on the coefficients if   is small. Infinitesimal variations _dc_ 1. . . , _dc n_ in the coefficients produce the variation _dλ_ according to the formula of calculus:

Since   we find

#### **PROBLEMS**

**1.** Show what happens when the method of iteration is applied to

What are the coordinates of the vectors _x k_ for _k =_ 760, 761, 762, 763 ?

**2.** Suppose that _A_ has a simple eigenvalue _λ i_ which is known to be approximately equal to −1.83. Form the matrix _B_ = ( _A_ \+ 1.83I)−1, and apply the method of iteration to _B_. What is the dominant eigenvalue of _B_ ? How can you compute _λ i_?

**3.** Suppose that the real matrix _A_ has distinct eigenvalues. Suppose that two of the eigenvalues, _λ i_ and   are known to be approximately equal to 1 ± _i_. Form the real matrix

How can the method of iteration be used to compute _λ j_ and   ?

**4.** Let  . Define the rate of convergence to be the reciprocal of the asymptotic value _v e_ for the number of iterations needed to reduce the error by the factor 1/ _e_. Show that, for the method of iteration,

**5.** Let _A_ be a real, symmetric, positive-definite matrix with the eigenvalues _λ_ 1 > _λ_ 2 > ... > _λ n_. Suppose that we have computed numerical values for _λ_ 1 and for a corresponding unit eigenvector _u_ 1. If _y_ ≠ a multiple of _u_ 1, form

The vector _x_ 0 is nonzero and is (apart from rounding error) orthogonal to _u_ 1. If there were no rounding error and if _u_ 1 were exact, show that the method of iteration, applied to _x_ 0, would produce the second eigenvalue, _λ_ 2. Explain why rounding error causes the method of iteration, applied to _x_ 0, to produce the dominant eigenvalue, _λ_ 1, again.

**6.** In formula (24), the denominator

equals zero if and only if the vectors _x_ and _y_ are dependent, as we saw in the proof of the Cauchy-Schwarz inequality. Why are the vectors _x_ and _y_ independent whenever formula (24) is applied in the method of iteration ?

**7.** Let _A_ be a real, symmetric matrix, with eigenvalues  .

Let _λ_ be any number satisfying the test for a real eigenvalue:

Prove that, for some eigenvalue _λ j_,

Thus,   represents an upper bound for the proportional error. (Expand _x_ in terms of unit orthogonal eigenvectors _u_ 1, . . . , _u n_.)

**8.** * Let _A_ be an _n_ × _n_ matrix with independent unit eigenvectors _u_ 1, . . . , _u n_. Let _T_ be the matrix with columns _u_ 1, . . ., _u n_. If

prove that, for some eigenvalue _λ j,_

Use the inequality

### **7.10 REDUCTION TO OBTAIN THE SMALLER EIGENVALUES**

In the last section we supposed that _A_ was an _n_ × _n_ real matrix with real and complex eigenvalues _λ_ l . . . , _λ n_. We showed how to compute either a dominant real eigenvalue, _λ_ 1, satisfying

or a dominant complex-conjugate pair,  , satisfying

_We will now show how to obtain from_ A _a reduced matrix_ R _whose eigenvalues are the remaining eigenvalues of_ A. The method of iteration for dominant eigenvalues, when applied to _R,_ yields one or two smaller eigenvalues of _A._ The matrix _R_ then can be reduced further to a matrix _R′,_ etc., until all the eigenvalues of the original matrix _A_ have been computed.

In the real case (1) the method of iteration yielded quantities _λ_ = _λ_ 1, _x_ , and _y_ = _Ax_ satisfying

apart from a negligible error. Suppose for the moment that

Form the nonsingular _n_ × _n_ matrix

We will show that

where the ( _n_ − 1) × ( _n_ − 1) matrix _R_ has the required property

The numbers * in the top row of (6) are irrelevant.

To prove (6), multiply (5) on the left by _A_. Since _Ax_ = _λ_ 1 _x_ and _Ae j_ = _a j_, the _j_ th column of _A_ , we find

The matrix _B_ = _T_ −l _AT_ satisfies the equation _TB_ = _AT_. If the columns of _B_ are _b_ 1, . . . , _b n_, then the columns of _TB_ are _Tb_ 1, . . . , _Tb n_. By (8), the equation _TB_ = _AT_ implies

Since _x_ is the first column of _T_ , the solutionof the equation _Tb_ l = _λ_ 1 _x_ is

Therefore, _B_ = _T_ −l _AT_ has the form (6). The identity (7) follows by taking the characteristic determinants of the matrices on both sides of (6):

Recalling the invariance (see Theorem 1, Section 4.2)

we find from (11) the identity

Division of both sides of (13) by _λ_ − _λ_ 1 yields the required result (7). Note that _I_ in the expression _λI_ − _R_ , is the identity of order _n_ − 1.

To compute _R_ , we verify that

Indeed, since _x_ 1 = 1, the matrix (14) times the matrix (5) equals _I_. Since _R_ comprises the elements in _i_ ,   in the product _T_ −1 _AT_ , we have

where _X_ consists of the rows   of _T_ −1, and _Y_ consists of the columns   of _T_. Explicitly,

Therefore,

which yields _R_ = ( _r ij_) in the explicit form

Now we shall eliminate the assumption that 1 = _x_ 1 = max | _x i_ |. This assumption was convenient because it insured that the matrix _T_ , denned by (5), was nonsingular and because it produced no large numbers _x_ _i_ \+ 1 in the final formula (18). Given A, _λ_ 1, and _x_ ≠ 0, we will find a new _A_ and a new _x_ satisfying _Ax_ = _λ_ 1 _x_ such that the new _A_ has the same eigenvalues as the old _A_ , and such that the new _x_ does satisfy 1 = _x_ 1 = max | _x i_ |. Let

Since _x_ ≠ 0, we have _x m_ ≠ 0. The vector _u_ satisfies

Let _P ij_ be the matrix which permutes components _i_ and _j_ of a vector. For example, if _n_ = 3,

_P ij_ is the matrix formed by interchanging rows _i_ and _j_ of the identity matrix. Note that

Note also that _AP ij_ is formed by interchanging _columns i_ and _j_ in _A_ ; _P ijA_ is formed by interchanging _rows i_ and _j_ in _A_.

Referring to (20), we see that _υ_ = _P_ l _m_ _u_ is a vector with maximal component _υ_ 1 = 1. Further,

But, by (21), _PAP_ = _P_ −l _AP_ , which has the same eigenvalues as _A_. Thus, we define

The new matrix _A_ and the new vector _x_ have all the required properties, including (4). Note that the new _A_ is formed from _A_ by interchanging rows 1 and _m_ and interchanging columns 1 and _m_. The reduced matrix _R_ is then computed directly from formula (18).

In the complex case (2), the method of iteration yielded real quantities _α, β, x, y_ = _Ax_ , and _z_ = _Ay_ satisfying

We know that _x_ and _y_ are independent because in the testing procedure of the preceding section, the numbers _α_ and _β_ were computed only after the failure of an attempt to express _y_ as a scalar multiple of _x_ ≠ 0. For the moment assume that _x_ and _y_ have the special forms

The coefficients _α, β_ satisfy

_We will find an_ (n − 2) × (n − 2) _matrix_ R _whose eigenvalues are λ_ 3, . . ., _λ n_.

Define the _n_ × _n_ matrix

Since _Ax_ = _y_ and _Ay_ = _z_ = − _βx_ − _αy_ ,

The matrix _B_ = _T_ −l _AT_ satisfies _TB_ = _AT_. If _b_ 1, ... are the columns of _B_ , then by (28)

By (27) we conclude that _b_ l and _b_ 2 have the forms

Therefore, _B_ = _T_ −l _AT_ has the form

The numbers * are irrelevant. Because of the zeros below the 2 × 2 block in the upper-left corner,

Because _B_ is similar to _A, B_ has the same characteristic determinant. Therefore (32) implies

The identity _λ_ 2 \+ _αλ_ \+ _β_ = ( _λ_ − _λ_ 1)( _λ_ − _λ_ 2) now gives

To compute _R,_ we verify that

Indeed, since _x_ and _y_ have the special forms (25), the matrix (35) times the matrix (27) equals _I_. Since _R_ consists of the elements in rows   and columns   of the product _T −lAT,_ we have

where _X_ consists of the rows   of _T_ −l, and _Y_ consists of the columns   of _T._ Explicitly,

Therefore,

which yields _R_ = ( _r ij_) in the explicit form

Suppose now that _x, y_ = _Ax_ , and _z_ = _Ay_ satisfy the dependence (24) but that _x_ and _y_ do _not_ have the special form (25). Then _x_ and _y_ are independent vectors satisfying

Here  _xy_ ] represents the _n_ × 2 matrix with columns _x_ and _y._ Let _N_ and _Q_ be nonsingular _n_ × _n_ and 2 × 2 matrices. By [(40),

Set

Suppose further that the vectors _x_ ′, _y_ ′ have the special form

In analogy to (27), let _T_ ′ be the _n_ × _n_ matrix

Equation (41) states that

where the 2 × 2 matrix _S_ is denned in (42). Note that

By analogy to (31), _B_ ′ = ( _T_ ′)−l _A_ ′ _T_ ′ has the form

Therefore,

But _B_ ′ is similar to _A_ ′, which is similar to _A_. Therefore,

Dividing (48) by ( _λ_ − _λ_ 1)( _λ_ − _λ_ 2), we find

Therefore, _R_ ′ is the required reduced matrix. By analogy to (39), we have the explicit form _R_ ′ = ( _r_ ′ _ij_ ), with

It remains only to find an _n_ × _n_ matrix _N_ and a 2 × 2 matrix _Q_ such that _N_  _xy_ ] _Q_ = [ _x_ ′ _y_ ′], where _x_ ′, _y_ ′ have the form [(43). Let

Define  . Define _N_ 1 = _P_ 1 _m_ , the matrix which permutes the first and wth components. Then we have the form

Let _Q_ 2 be a 2 × 2 matrix which subtracts _η_ times the first column from the second column:

Then, by (53), we have the form

Let  ; this maximum is positive because the two columns (55) remain independent. Let _N_ 2 = _P_ 2 _k_ , and let _N_ 3 divide row 2 by _ζ k_. Then we have the form

Explicitly, _N_ 3 = diag  . Let _Q_ 3 subtract _ω_ times column 2 from column 1:

Then we obtain _x_ ′, _y_ ′

There is no need to obtain _N_ = _N_ 3 _N_ 2 _N_ 1 and _Q_ = _Q_ 1 _Q_ 2 _Q_ 3 explicitly, but we must obtain _A_ ′ = _NAN_ −1 explicitly to compute the reduced matrix _R_ ′ from (51). We have

since  . Formula (59) gives the following procedure. Begin with _A._ Interchange columns 1 and _m_. Interchange rows 1 and _m_. Interchange columns 2 and _k._ Interchange rows 2 and _k._ Divide row 2 by _ζ k_. Multiply column 2 by _ζ k_. The result is _A_ ′. Since the vectors _x_ ′, _y_ ′ were given by (58), we can now compute the reduced matrix from (51).

Although the analysis has been long, the explicit computations (58), (59), (51) are very brief. Therefore, double precision can be used with a negligible increase in computing time. In any case, rounding error is reduced by the choice of the maximal pivot-elements _x m_ and ζ _k_.

#### **PROBLEMS**

**1.** Let

This matrix has the simple eigenvalue _λ_ 1 = 4, with the eigenvector _x_ = col (1, 1, 1). Find the 2 × 2 reduced matrix _R._

**2.** Let

This matrix has the simple eigenvalue _λ_ l = −1, with the eigenvector _x_ = col (0, 3, 10). Find the 2 × 2 reduced matrix _R._

**3.** Let

Verify that _A_ 2 _x_ − 4 _Ax_ \+ 5 _x_ = 0. Compute two complex-conjugate eigenvalues. Using formula (51), compute the l × l reduced matrix _R._ First perform the transformations (58) and (59).

**4.** Let _x_ and _Ax_ be independent; let _x, Ax,_ and _A_ 2 _x_ be dependent:

If _λ_ 2 \+ _αλ_ \+ _β_ ≡ ( _λ_ − _λ_ 1)( _λ_ − _λ_ 2), show that ( _A_ − _λ_ 2 _I_ ) _x_ is an eigenvector belonging to _λ_ 1

**5.** Let

Verify that _x, Ax,_ and _A_ 2 _x_ are dependent. Compute two complex eigenvalues, and form the 2 × 2 reduced matrix, _R._ Then compute the other two eigenvalues of _A_.

### **7.11 EIGENVALUES AND EIGENVECTORS OF TRIDIAGONAL AND HESSENBERG MATRICES**

Some methods for computing the eigenvalues of an _n_ × _n_ matrix _A_ depend upon finding the coefficients _c_ _j_ of the characteristic polynomial

The eigenvalues are then found by some numerical method for computing the roots of polynomial equations.

In the method of Householder and Bauer, to be presented in the next section, given _A_ , we construct a unitary matrix _U_ such that the similar matrix _U* AU_ has the special form

This is the _Hessenberg form_ , with

In particular, if _A_ is Hermitian, then _H_ is Hermitian. _A Hermitian Hessenberg matrix_ H _is tridiagonal, with_  .

The reduction (2) is useful if there is an easy way to find the coefficients of the characteristic polynomial of a Hessenberg matrix, since the identities _U*AU_ = _H_ , _U*U_ = _I_ imply

_The purpose of this section is to show how to find the coefficients_ cj _of the characteristic polynomial of a Hessenberg matrix_ H. As a by-product, we shall show how to find an eigenvector belonging to a known eigenvalue of _H_. We also will show, in the Hermitian case, that _the eigenvalues can be located in intervals_ without any numerical evaluation of the roots of the characteristic equation.

Given _H_ = _H_ _n_ , construct the sequence of Hessenberg matrices

Let Δ1 = det _H_ 1, . . . , Δ _n_ = det _H_ _n_. We will develop a recursion formula for these determinants.

**E XAMPLE 1.** For _n_ = 5 we have

We shall expand Δ5 = det _H_ 5 by the last row.

Thus, if we define Δ0 = 1,

In the general case, expanding Δ _n_ = det _H_ _n_ by the last row, we find

To find the characteristic polynomial  , we only need to replace the diagonal elements _h vv_ by _h vv_ − λ. Then (7) yields, for  ,

where   and  . _This is a formula for computing, successively_ ,  , . . . . If _H n_ is tridiagonal, (8) takes the simpler form

Next we shall show that, _if_ λ _is an eigenvalue of_ Hn, _an eigenvector_ x = col (x1, . . . , xn) _is given by_

_provided that at least one of these numbers_ xi _is nonzero_. Since _x_ 1 = ± _α_ 1 . . . _α_ _n_ −1, we have at least _x_ 1 ≠ 0 if all _α_ _v_ ≠ 0.

To show that (10) provides an eigenvector, we must show that, if  , then

If _y_ = ( _H_ _n_ − λ _I_ ) _x_ , then

which equals  , by (8). For _k_ < _n_ we have

But this expression equals 0, as we see by replacing _n_ by _k_ in the recursion formula (8).

_If_ H _is Hermitian, with all α_ j ≠ 0, _we will show how the eigenvalues can be located in intervals_. Since  , the polynomials  , . . . ,   defined by (9) satisfy

Let _a_ < λ < _b_ be a given interval. We suppose that neither _a_ nor _b_ is a zero of any of the polynomials  . _We will count the number of zeros of  in the interval_ (a, b). Note the following properties:

(i)   in (a, b). [In fact,  .]

(ii) _If_  , _then_   _and_   _are nonzero and of opposite signs_.

_Proof._ By (13), if either adjacent polynomial,   or  , is zero, the other one must also be zero. But then   and   have a common zero, and the recursion formula shows that  , etc. Finally, we could conclude  , which is impossible. Therefore,   implies both   and  . By (13),

Therefore,   and   have opposite signs.

(iii) _As_ λ _passes through a zero of_  , _the quotient_   _changes from positive to negative_.

_Proof:_ By (ii), the adjacent polynomials   and   cannot have a common zero. By the inclusion principle (Section 6.4), the zeros _μ j_ of   separate the zeros _λ j_ of  :

Furthermore,

Therefore,   changes sign exactly once, at the pole _λ_ = _μ_ _j_ , between consecutive zeros _λ_ _j_ +1, _λ_ _j_ ; and we have Fig. 7.6 as a picture of the algebraic signs of  . We see that the sign changes from + to − as _λ_ crosses each _λ_ _j_.

**Figure 7.6**

Sequences of polynomials   with properties (i), (ii), (iii) are called _Sturm sequences_. They have this remarkable property:

**Theorem 1.** _Let  be Sturm sequence on the interval_ (a, b). _Let σ_ ( _λ_ ) _be the number of sign changes in the ordered array of numbers . Then the number of zeros of the function   in the interval_ (a, b) _equals σ_ (b) − _σ_ (a).

**E XAMPLE.** Let

If   and   for  , then

For _λ_ = 0 we have the signs

Hence, _σ_ (0) = 1. For _λ_ = 10 we have the signs

Hence, _σ_ (10) = 3. The number of zeros of   in the interval (0, 10) is, therefore, _σ_ (10) − _σ_ (0) = 3 − 1 = 2.

_Proof of Sturm's Theorem_. The function _σ_ ( _λ_ ) can change only when _λ_ passes through a zero of one of the functions  . By property (i),   never changes sign in ( _a_ , _b_ ). If  , consider the triple

If  , property (ii) states that   and   have opposite signs. Therefore, the triple (14) yields exactly one sign change for _all λ_ in the neighborhood of _λ_ 0. Assume  . Then _σ_ (λ) cannot change as _λ_ crosses _λ_ 0.

If _λ_ crosses a zero _λ_ _v_ of  , property (iii) states that the signs of

change from + + to + −, or − − to + +. Therefore, exactly one sign change is added in the sequence of ordered values  . In other words, _σ_ (λ) increases by 1. In summary, _σ_ (λ) increases by 1 each time λ crosses a zero of  ; otherwise, _σ_ (λ) remains constant. Therefore, _σ_ ( _b_ ) − _σ_ ( _a_ ) is the total number of zeros of   in the interval ( _a_ , _b_ ).

In practice, Sturm's theorem can be used with Gershgorin's theorem (Section 6.8). For tridiagonal _H_ = _H*_ , we know from Gershgorin's theorem that every eigenvalue of _H_ satisfies one of the inequalities

where _h_ 01 = _h_ _n_ , _n_ +1 = 0. The union of the possibly-overlapping intervals (15) is a set of disjoint intervals

These intervals can be subdivided, and Sturm's theorem can be applied quickly to each of the resulting subintervals ( _a_ _iα_ , _b_ _iα_ ).

When Sturm's theorem is used numerically to locate the eigenvalues of tridiagonal _H_ = _H_ *, the coefficients of the characteristic polynomials   should _not_ be computed. Instead, the recursion formula (13) should be used _directly_ to evaluate successively the numbers   and hence the integer _σ_ (λ).

In the proof of Sturm's theorem, we assumed that

We can show that _it was only necessary to assume_

If (18) is true but (17) is false, then for all sufficiently small   > 0 we have

By what we have already proved, the number of zeros of   in ( _a_ \+  , _b_ −  ) is _σ_ ( _b_ −  ) − σ( _a_ \+  ). But

because   has no zeros for  . Thus, the number of zeros of   in ( _a_ , _b_ ) equals the number of zeros of   in ( _a_ \+  , _b_ −  ), which equals _σ_ ( _b_ ) − _σ_ ( _a_ ).

For a precise computation of the eigenvalues, Newton's method should be used after the eigenvalues have been separated and approximated by the theorems of Gershgorin and Sturm. Newton's method for computing the real roots of an equation   is discussed in texts on numerical analysis. This method converges extremely rapidly if the first approximation is fairly accurate.

#### **PROBLEMS**

**1.** Using the recurrence formula (8), evaluate det ( _H_ − _λI_ ) for the _n_ × _n_ Hessenberg matrix

**2.** Evaluate successively

**3.** * Let

Define Δ0 = 1, Δ1 = _c_ 1,

Prove that

Show that [(7) implies   as in Problem 2, evaluate Δ _n_ for all _n_.

**4.** Using formula (10), calculate an eigenvector for the matrix

belonging to the eigenvalue _λ_ = 0.

**5.** Let _H_ be a Hessenberg matrix, as in formula (2). Show that _H_ is similar to a Hessenberg matrix _H′_ with   if   if _α j_ = 0. (Form  , where A is a suitable diagonal matrix.)

**6.** Let

Let  . Using the recursion formula (13), calculate numerical values for the   for λ = 0 and for λ = 4. How many eigenvalues of _A_ lie between 0 and 4?

**7.** * Let _H n_ be the _n_ × _n_ matrix

(The matrix _A_ in the last problem is _H_ 5). Let   and, for  , let  . I Making the change of variable _λ_ = 2 − 2 cos _θ_ , prove that

Hence find all the eigenvalues of _H n_. Use the recursion [formula (13).]

### **7.12 THE METHOD OF HOUSEHOLDER AND BAUER**

Let _A_ be a real, symmetric _n_ × _n_ matrix. We will show how _A_ can be reduced to tridiagonal form by _n_ − 2 real, unitary similarity-transformations. For _n_ = 5 the process is illustrated as follows:

From the tridiagonal form, the characteristic polynomial can be found by the recursion formula (9) of the preceding section, or the eigenvalues can be located in intervals by Sturm's theorem (Theorem 1, Section 7.11).

If _A_ is real but not symmetric, the method will produce zeros in the upper right corner but not in the lower left corner. For _n_ = 5, the process would appear as follows:

The final matrix is in the Hessenberg form, and the characteristic polynomial can be found by the recursion formula (8) of the last section. If _A_ is complex, the method is modified in the obvious way, by replacing transposes _x T_ by their complex conjugates _x*._

In the real, symmetric case, the first transformation _A_ → _B_ = _U TA U_ is required to produce a matrix of the form

If all _a_ 1 _k_ already  , we set _B_ = _A_. Otherwise, we will use a matrix of the form

Explicitly, if the real column-vector _u_ has components _u i_, then _U_ = (δ _ij_ − 2 _u iuj_). Since we require _u_ to have unit length, _U_ is unitary.

_U_ is also real and symmetric: _U T_ = _U_. The requirements

place _n_ − 1 constraints on the _n_ -component vector _u_. Since there is one degree of freedom, _we set u_ 1 = 0. Note that _b_ l _k_ = 0 _implies b_ _k_ l = 0, since _B_ is symmetric.

The first row of _B_ = _UAU_ is the first row of _UA_ times _U_. Since _u_ 1 = 0, the first row of _U_ = _I_ − 2 _uu T_ is just (1,0, . . ., 0). Therefore, the first row of _UA_ is just the first row of _A._ Hence, the first row of _B_ is

Since _U_ = _I_ − 2 _uu T_, we have _a TU_ = _a T_ − 2( _a Tu_) _u T_, or

The unknowns _u_ 2, . . . , _u n_ must solve the _n_ − 1 equations

To solve these equations, we observe that the rows _a T_ and _b T_ = _a TU_ must have equal lengths: _b Tb_ = _a TUUTa_ = _a Ta._ Thus, we require

By (8), b11 = a11. Then (11) takes the form

If  , equations (8) and (12) yield

We now multiply (13) by _u 2_, (9) by _u j_ ( _j_ = 3, . . . , _n_ ), and add to obtain

or

or, since we require _u Tu_ = 1,

This is a relationship between the unknown inner product _a Tu_ and the unknown component _u 2_. Using (14) in the identity (13) for _b_ l2, we find

We have _α_ > 0 because at least one  . Therefore, (15) has the positive solution

_We choose the sign_ ± _so that_  . Since _α_ > | _a_ 12 |, we have    . Having computed _u_ 2, we compute _u_ 3. . . , _u n_ from (9) and (14).

**Exercise** _If_ ± _is chosen so that_  , _and if we define_

_verify directly that the numbers_ u2, . . . , un _defined by_ (16) _and_ (17), _solve the required equations_ (9) _and_ (10).

We now transform _A_ into _B_ = _UAU_ , with _b_ l _k_ = _b_ _k_ l = 0 for  . If _n_ > 3, we wish next to transform _B_ into a matrix _C_ = _VBV_ with

We shall require _V TV_ = _I_ and _V_ = _V T_.

Observe the matrix _B_ in (3). We can partition _B_ as follows:

where _A_ ′ is the ( _n_ − 1) × ( _n_ − 1) matrix  . By analogy to (16) and (17), we can reduce _A_ ′ to the form

by a transformation _B_ ′ = _U_ ′ _A_ ′ _U_ ′, where

If _A_ ′ has not already the form (21), we simply define

and define, in analogy to (16) and (17),

The sign ± is chosen to make  .

We now assert that _the required matrix_ V _has the form_

_Proof:_ Since  , the matrix _V_ has the form

The first row of _VBV_ is the first row of _V_ times _BV._ Since the first row of _V_ is (1, 0, . . . , 0), the first row of _VBV_ is just the first row of _B_ times _V_ , or

Here we have used the form (25) for _V._ Thus, _the first row and_ , by symmetry, _the first column are unaffected._ The partitoned form (24) now shows that

But _U′A′U′ = B′_ of the form (21). Therefore, _VBV_ = _C_ of the required form

where _A_ ″ is an ( _n_ − 2) × ( _n_ − 2) symmetric matrix.

We note parenthetically that _V_ in (24) has the form

where _υ_ is the unit vector col  .

The process now continues. Let _U_ ″ be the ( _n_ − 2) × (n − 2) matrix of the form _I_ ″ − 2 _u_ ″( _u_ ″) _T_ which reduces _A_ ″ to the form

We then define

Since  , this matrix has the form

Therefore, as (28) shows, the first two rows and the first two columns are unaffected by the transformation _C_ → _WCW_. Therefore,

After _k_ reductions we obtain a symmetric _n_ × _n_ matrix, say _Z_ = ( _z ij_), with

We call _A_ ( _k_ ) the _lower-right part_.

If _k_ < _n_ − 2, we will reduce _Z_ further. If at least one of the numbers   is nonzero, we form   from (16) and (17), replacing _A_ by _A_ ( _k_ ), and _n_ by _n_ − _k_. Setting   we have the form

where

If we define the _n_ × _n_ matrix

we must show that the first _k_ rows and the first _k_ columns of _Z_ are unaffected by the transformation _Z_ → _MZM_ = _Z_ ′. Now

Let   Then by (38),   and

By (34),  . Therefore, (39) yields

But (38) shows that

Then (40) implies   if  . By symmetry, we also have   if  .

Finally, we must show that _Z_ ′ = _MZM_ implies

Now the form (38) shows that

Therefore,

But (44) is equivalent to (42), since

COMPUTING ALGORITHM

The preceding three paragraphs give a rigorous justification of the method. In practice, the matrix _M_ in (38) is not formed explicitly, nor is the matrix _U_ ( _k_ )

At each stage _k_ = 0, 1, . . . , _n_ − 3 our task is to replace the ( _n_ − _k_ ) × ( _n_ − _k_ ) matrix _A_ ( _k_ ) by the matrix

_The number θ is unchanged_ =  . If, by chance,

then no computation is performed at stage _k_ , since _A_ ( _k_ ) already has the form of the right-hand side of (46). Otherwise, _we do compute the numbers_   by (16) and (17), replacing _A_ by _A_ ( _k_ ), and _n_ by _n_ − _k_. Dropping the superscript _k_ , we observe that

Define the quadratic form

_Compute_ Au = b, i.e., _compute_

By (48), the _i_ , _j_ -component of _U_ ( _k_ ) _A_ ( _k_ ) _U_ ( _k_ ) is

_Replace_   _by_

_Replace_   _by_ 0, . . . , 0. _For_   _replace_   _by the following equivalent of_ (51):

_where_

_The components with_ i > j _are now formed by symmetry,_

COUNT OF OPERATIONS

We will count the number of multiplications and the number of square roots. At stage _k_ = 0,1, . . . , _n_ − 3, the calculations

require 2 square roots and approximately 2( _n_ − _k_ ) multiplications. To form _b_ in (50) requires ( _n_ − _k_ )( _n_ − _k_ − 1) multiplications. The calculation of _c_ requires _n_ − _k_ − 1 multiplications. The calculations (53) require

which is equal to ( _n_ − _k_ ) ( _n_ − _k_ − 1) multiplications. In summary, at stage _k_ there are

Summing for _k_ = 0, . . . , _n_ − 3, we find a total of

Thus, there are very few multiplications. After all, just to square an _n_ × _n_ matrix requires _n_ 3 multiplications. Since the number of multiplications is small, it will usually be economically feasible to use double precision.

#### **PROBLEMS**

**1.** Reduce the matrix

to tridiagonal form by the method of Householder and Bauer.

**2.** Reduce the matrix

to Hessenberg form by the method of Householder and Bauer.

**3.** * Reduce the matrix

to tridiagonal form by the method of Householder and Bauer.

**4.** * According to formula (55), the H-B reduction of an _n_ × _n_ symmetric matrix requires "about" 2 _n_ 3/3 multiplications. What is the exact number of multiplications?

**5.** If _A_ is real but not symmetric, and if _A_ is to be reduced to Hessenberg form by the H-B method, how should formulas (50)–(53) be modified?

**6.** If a real matrix _A_ is reduced to a tridiagonal or Hessenberg matrix _X_ by the H-B method, show that  . (This identity can be used as a check of rounding error.)

### **7.13 NUMERICAL IDENTIFICATION OF STABLE MATRICES**

In the theory of automatic control, in circuit theory, and in many other contexts the following problem arises: _Given an_ n × n _matrix_ A, _determine whether all of its eigenvalues lie in the left half-plane_ Re λ < 0. The matrix A is called _stable_ if all of its eigenvalues lie in the left half-plane. The importance of this concept is shown by the following theorem:

**Theorem 1.** _An_ n × n _matrix_ A _is stable if and only if all solutions_ x(t) _of the differential equation_

_tend to zero as_ t → ∞.

_Proof_. Suppose that _A_ has an eigenvalue _λ_ with Re  . If _c_ is an associated eigenvector, then the solution

does not tend to zero as _t_ → ∞ because  .

Suppose, instead, that A is stable. Let _λ_ 1 . . . , _λ_ s be the different eigenvalues of _A_ , with multiplicities _m_ 1 . . . , _m_ s. In formula (17) of Section 5.1, we showed that every component _x v_( _t_ ) of every solution _x_ ( _t_ ) of the differential equation _dx_ / _dt_ = _Ax_ is a linear combination of polynomials in _t_ times exponential functions exp, _λ it_. Since all _λ i_ have negative real parts, every such expression tends to zero as _t_ → ∞. This completes the proof.

One way to show that a matrix is stable is to compute all its eigenvalues and to observe that they have negative real parts. But the actual computation of the eigenvalues is not necessary. By results of _Routh and Hurwitz_ , it is only necessary to compute the coefficients _c i_ of the characteristic polynomial

Assume that _A_ is real, andhence that the _c i_ are real. Form the fraction

If _c_ 1 ≠ 0, this fraction can be rewritten in the form

where α1 = 1/ _c_ 1 and

with

Similarly, we write

and so on until we obtain the full continued fraction

**Theorem 2.** _The zeros of the polynomial λ_ n \+ _c_ 1 _λ_ n−1 \+ ... + _c_ 0 _all lie in the left half-plane if and only if the coefficients α_ 1 _in the continued fraction_ (7) _are all positive._

**E XAMPLE 1.** Consider the polynomial

We have

with

The full continued fraction is

Here

Since all _α i_ are positive, Routh's theorem states that the zeros _λ v_ of the polynomial (8) are in the left half-plane. In fact, the zeros are

We will not prove Routh's theorem, which is not a theorem about matrices, but a theorem about polynomials. A proof and a thorough discussion appear in E. A. Guillemin's book, _The Mathematics of Circuit Analysis_ (John Wiley and Sons, Inc., 1949) pp. 395–407. _If the_ n × n _matrix_ A _is real, and if_ n _is small, Routh's theorem provides a practical criterion for the stability of the matrix_ A. But if _n_ is large, Routh's theorem is less useful because it is hard to compute accurately the coefficients of the characteristic polynomial. If the coefficients _c i_ are very inaccurate, there is no use in applying Routh's theorem to the wrong polynomial.

A second criterion, which applies directly to the matrix A, and which does not require that _A_ be real, is due to _Lyapunov._

**Theorem 3.** _The_ n × n _matrix_ A _is stable if and only if there is a positive definite Hermitian matrix_ P _solving the equation_

If _n_ is large, this theorem is hard to use numerically. The system (9) presents a large number, _n_ ( _n_ \+ l)/2, of linear equations for the unknown components  . One must then verify that _P_ is positive definite by the determinant criterion (see Section 6.5)

Because Lyapunov's theorem has great theoretical interest, we shall give the proof.

_Proof._ First assuming _PA_ \+ _A*P_ = _–I,_ we shall prove that _A_ is stable. By Theorem 1, it will be sufficient to show that every solution of the differential equation _dx/dt_ = _Ax_ tends to zero as _t_ → ∞. We have, for the solution _x_ ( _t_ ),

Since _P_ is positive definite, there is a number _ρ_ > 0 for which

Then (11) yields

Therefore,

If _x_ (0) = _b,_ we conclude that  , and hence

But   for some _σ_ >0. Therefore,

which shows that _x_ → 0 as _t_ → ∞.

Conversely, supposing that _A_ is stable, we shall construct the positive-definite matrix _P_ solving _PA_ \+ _A*P_ = – _I_. Let the _n_ × _n_ matrix _X_ ( _t_ ) solve

As in the proof of Theorem 1, we know that every component of the matrix _X_ ( _t_ ) is a linear combination of polynomials in _t_ (possibly constants) times exponential functions exp   where Re   We bear in mind that the adjoint matrix _A*_ has the eigenvalues   if _A_ has the eigenvalues _λ v_. Taking adjoints in (17), we find

Now the product _X_ ( _t_ ) _X*_ ( _t_ ) satisfies

Define the integral

This integral converges because every component of the integrand has the form

which is a linear combination of polynomials in _t_ times decaying exponential functions. By integrating equation (19) from _t_ = 0 to _t_ = ∞, we find

The left-hand side, – _I_ , appears because _XX*_ equals _I_ when _t_ = 0 and equals zero when _t =_ ∞. The matrix _P_ is positive definite because the quadratic form belonging to a vector _z_ ≠ 0 equals

This completes the proof.

_Lyapunov's theorem shows that, if a matrix_ A _is stable, then there is a family of ellipsoidal surfaces_ (Pz, z) = const _relative to which every solution of_ dx/dt = Ax _is constantly moving inward._ In fact, (11) shows that

A PRACTICAL NUMERICAL CRITERION

Let us suppose that _A_ is an _n_ × _n_ matrix, not necessarily real, with _n_ large. We will give a criterion for the stability of _A_ which

(a) does not require explicit computation of the eigenvalues;

(b) does not require explicit computation of the coefficients of the characteristic polynomial; and

(c) does not require the solution of systems of _N_ linear equations with

If _A_ has eigenvalues _λ_ 1, . . . , _λ_ n, and if _τ_ is any positive number, the numbers

all lie in the unit circle | _μ_ | < 1 if and only if all the numbers _λ j_ lie in the left half-plane. This is evident from Figure 7.7.

**Figure 7.7**

The absolute value | _μ_ j | equals the distance of _λ j_ from – _τ_ divided by the distance of _λ j_ from _τ_. In the figure Re _λ j_ > 0; therefore | _μ_ j | > 1.

_The numbers μ j are the eigenvalues of the_ n × n _matrix_

unless _τ_ = some _λ j_. Proof:

Assume that _τ_ > 0 has been chosen. The problem now is to determine whether all the eigenvalues _μ j_ of the matrix _B_ lie inside the unit circle | _μ_ _j_ | < 1. By Theorem 3 in Section 4.9, all | _μ_ _j_ | < 1 if and only if

In principle, we could form the powers _B k,_ but we prefer not to do so because each multiplication of two _n_ × _n_ matrices requires _n 3_ scalar multiplications.

We prefer to form the iterates _B_ _k_ _x_ (0) = _x_ (k), where _x_ (0) is some initial vector. As the vectors _x_ (0), _x_ (l), _x_ (2), . . . are computed, we look at the successive lengths squared:

If these numbers appear to be tending to zero, we infer that _A_ is stable; if these numbers appear to be tending to infinity, we infer that _A_ is unstable; if the numbers || _x_ ( _k_ ) ||2 appear to be tending neither to 0 nor to ∞, we infer that _A_ is stable or unstable by a very small margin. We will justify this procedure in Theorem 4.

If _A_ is a stable _normal_ matrix—e.g., a stable Hermitian matrix—the numbers (28) will form a monotone decreasing sequence. We see this from the expansion

of _x_ (0) in the unit orthogonal eigenvectors _u j_ of _A_. The vectors _u j_ are also eigenvectors of _B_. Then

where all | _μ j_ | < 1. If _A_ is not a normal matrix, the sequence (28) will not usually be monotone.

How large should the number _τ_ > 0 be chosen? If _τ_ is too small or too large, all of the eigenvalues

of _B_ are near ±1. If any _λ j_ has Re _λ j_ > 0, we would like the modulus | _μ j_ | to be as large as possible. We assert that, if Re _λ_ > 0, then

_Proof_. If λ = _ρe iθ_, with _| _θ_ |< π_/2, we must show that

This is true if

or if

which is true, with equality only if _τ_ = _ρ_. If _A_ is given, a reasonable first guess for the modulus, _ρ_ , of an eigenvalue is

We find this value by summing

for _i_ = 1, . . . , _n_ , and by falsely replacing all _a ij_ by | _a ij_ | and all _x i_ by 1. _The number τ defined by_ (30) _is of the right general size_.

**Lemma.** _Let_ B _be an_ n × n _matrix. Let_ p _be a principal vector of grade  belonging to the eigenvalue μ. Then for large_ k _we have the asymptotic forms_

_where_ v _is an eigenvector belonging to _μ_ and where the remainder_ r(k) _equals_ 0 _if_ g = 1 _or equals_

_Proof_. By the notation (32) we mean that, for some constant _γ_ > 0,

If _g_ = 1, then _p_ is an eigenvector, _p = υ_ , and _r_ ( _k_ ) = 0 in (31). If  , we have for

In the expansion by the binomial theorem, we have

for the principal vector _p_. The vector

is an eigenvector because _p_ has grade _g_. Since _g_ is fixed as _k_ → ∞, we have

For the other terms in (33) we have   in the formula

The last three formulas yield the required asymptotic form (31).

In the following theorem, we shall use the phrase "with probability one," and we wish now to define the sense in which this phrase will be used. Consider an arbitrary nonzero _n_ -component vector _x_. Let _B_ be an _n_ × _n_ matrix with the different eigenvalues _μ_ 1 . . . , _μ s_ with multiplicities _m_ 1 . . . , _m s_, where Σ _m i_ = _n_. As we showed in Section 5.2, the vector _x_ has a unique expansion

in principal vectors _p_ ( _j_ ) belonging to the eigenvalues _μ j_. In the last paragraph of Section 5.3 we concluded that the space of principal vectors _p_ ( _j_ ) has dimension _m j_. Let _Z j_ be the linear space of vectors _x_ for which _p_ ( _j_ ) = 0 in the representation (36). Then the dimension of _Z j_ is _n_ − _m j_ < _n_. Thus, _Z j_ has _n_ -dimensional measure zero. The set _Z_ of vectors _x_ for which at least one _p_ ( _j_ ) = 0 in the expansion (36), is the union

of the sets _Z j_, all of which have measure zero. Therefore, _Z_ has measure zero. In this sense we say that, _with probability one_ , all of the principal vectors _p_ ( _j_ ) are nonzero in the expansion (36) of a random vector _x_.

**Theorem 4**. _Let_ **B** _be an_ n × n _matrix. Let_ x = x(0) _be a random vector_.

_Let_

_x_ (1) = _Bx_ (0), _x_ (2) = _Bx_ (1), _x_ (3) = _Bx_ (2), . . .

Let _μ_ 1, _μ_ 2, . . . _be the eigenvalues of_ B

Case 1. _if all_ | _μ j_ | < 1, _then_ x(k) → 0 _as_ k → ∞.

Case 2. _If all_  , _and if at least one_ | _μ j_ | = 1, _then_ with probability one _the sequence_ x(k) _does not tend to zero_.

Case 3. _If at least one_ | _μ j_ | > 1, _then_ with probability one || x(k) || → ∞ _as_ k → ∞.

_Proof_. In Case 1 we know that _B k_ → 0 as _k_ → ∞, and therefore

Let _μ_ 1, . . . , _μ_ s be the different eigenvalues of _B_ , with multiplicities _m_ 1 . . . , _m s_. Let

where Σ _m i_ = _n_ and  . Form the expansion (36) of _x_ as a sum Σ _p_ ( _j_ ) of principal vectors of _B_. With probability one _all p_ ( _j_ ) ≠ 0. In other words, with probability one every _p_ ( _j_ ) in the expansion _x_ = Σ _p_ ( _j_ ) has grade  . Then, by the preceding lemma,

where _υ_ (1), . . . , _υ_ ( _s_ ) are eigenvectors corresponding to _μ_ 1 . . . , _μ_ _s_. Let

Let _J_ be the set of integers _j_ such that   and such that _g j_ = _g_. Then (38) yields

where _ρ_ = | _μ_ l | = . . . = | _μ r_ |. By the notation o( _K_ _g_ −1 _ρ k_) we mean any function of _k_ such that

For all _j_ ∈ _J_ we have | _μ j_ | = _ρ_. Let

where the angles _θ j_ are distinct numbers in the interval  . In Cases 2 and 3 we assume  . From (40) we have

Eigenvectors _υ_ ( _j_ ) belonging to different eigenvalues _μ j_ are linearly independent. Therefore, the minimum length

is a positive number _δ_ > 0. Letting _α j_ = exp ( _ikθ j_) in (43), we find

In Case 2 we have _ρ_ = 1, and || _x_ ( _k_ ) || → ∞ if g > 1. If _g_ = 1 and _ρ_ = 1, then (43) shows that _x_ ( _k_ ) remains bounded but (45) shows that _x_ ( _k_ ) does not tend to zero. In Case 3 we have _ρ_ > 1, and (45) shows that || _x_ ( _k_ ) || → ∞.

#### **PROBLEMS**

**1.** Let _A_ be an _n_ × _n_ matrix. Let   and _ψ_ ( _λ_ ) be polynomials with real or complex coefficients. Suppose that _ψ_ ( _λ j_) ≠ 0 for all eigenvalues _λ j_ of _A_. Prove that the _n_ × _n_ matrix _ψ_ ( _A_ ) has an inverse. Then prove that the eigenvalues of the matrix   are   . (First prove the result for triangular matrices _T_. Then use the canonical triangularization _A_ = _UTU_ *.)

**2.** Find the Lyapunov ellipses ( _Px, x_ ) = const for the stable matrix

**3.** Let _A_ be any stable _n_ × _n_ matrix and let _Q_ be any _n_ × _n_ positive-definite Hermitian matrix. Show that there is a positive-definite matrix _P_ solving the equation − _Q_ = _A*P_ \+ _PA_.

**4.** Let _A_ be any stable _n_ × _n_ matrix. Let _C_ be an arbitrary _n_ × _n_ matrix. Show that the equation − _C_ = _A*M_ \+ _MA_ has a solution _M_. Since _every_ equation − _C_ = _A*M_ \+ _MA_ has a solution, _M_ , deduce that the particular equation − _I_ = _A*P_ \+ _PA_ has a _unique_ solution, _P_.

**5.** Let _λ_ 3 \+ _c_ 1 _λ_ 2 \+ _c_ 2 _λ_ \+ _c_ 3 have real coefficients. According to the Routh-Hurwitz criterion, what inequalities are necessary and sufficient for the zeros of the polynomial to have negative real parts?

**6.** For the stable matrix _A_ in Problem 2, compute the number T in formula (30). Compute the matrix _B_ in formula (26). What are the eigenvalues of _B_?

**7.** Let

Compute the number _τ_ and the matrix _B_. For which vectors _x_ do the iterates _B kx_ tend, in length, to infinity as _k_ → ∞? What is the probability, for a random vector _x_ , that || _B kx_ || → ∞ as _k_ → ∞?

### **7.14 ACCURATE UNITARY REDUCTION TO TRIANGULAR FORM**

In the _QR_ method of computing eigenvalues, we shall need an accurate way of reducing an _n_ × _n_ matrix _A_ to a right-triangular matrix _R_ by transformation

Here we assume that _A, U_ , and _R_ are real or complex matrices, with

In principle, we could perform this transformation by the Gram-Schmidt process described in Section 4.5. If the columns _a_ 1, . . . , _a n_ of _A_ were independent, we could form unit vectors _υ_ 1, . . ., _υ n_ such that _a k_ would be a linear combination of _υ_ 1, . . . , _υ k_.

In other words.

or

_A_ = _VR_

Then we could take _U_ = _V_ * in (1). Unfortunately, the Gram-Schmidt process has been found to be highly inaccurate in digital computation. This inaccuracy is demonstrated in J. H. Wilkinson's book.

In this section, we shall show how to perform the reduction _UA_ = _R_ by a sequence of reflections

where each _U j_ has the form _I_ − 2 _ww_ *, or where _U j_ = _I_. Here _w_ is a unit vector depending on _j_. The matrices _U j_ are both unitary and Hermitian:

The matrix _U j_ will produce zeros below the diagonal in column _j_ ; and _U j_ will not disturb the zeros which shall already have been produced in columns 1, . . . , _j_ – 1. Thus, if _n_ = 4,

Let _A_ 1 = _A_ , and for _j_ > 1 let

_A_ _j_ = _U_ _j_ − 1. . . _U_ 1 _A_

Assume that _A j_ has zeros below the diagonal in columns 1, . . . , _j_ − 1. Let the _j_ th column of _A_ _k_ have the form

Let

If _β_ = 0, then _A j_ already has zeros below the diagonal in column _j_ , and we set _U j_ = _I_.

If _β_ > 0, we look for _U j_ in the form

We shall have

For this vector to have zeros in components _j_ \+ 1, . . . , _n_ , we must have _w_ _j_ +1, . . . , _w n_ proportional to _c_ _j_ +1, . . . , _c n_, since 2( _w_ * _c_ ) is simply a scalar.

Thus, we look for _w_ in the form

Then

and _U jC_ will have zeros in components _j_ \+ 1, . . . , _n_ if

For _w_ to be a unit vector, we must have

Elimination of | _λ_ |2 from (11) and (12) gives

If _c j_ = | _c j_ | exp ( _iγ j_), we will look for _μ_ in the form _μ_ = | _μ_ | exp ( _iγ j_). Then (13) becomes

which has the positive solution

Then

Now (11) yields

and hence

Then _U j_ = _I_ – 2 _ww_ *.

We must show, finally, that multiplication by _U j_ does not disturb the zeros below the diagonal in columns 1, . . . , _j_ − 1. If _z_ is any of the first _j_ − 1 columns of _A j_, then _z j_ = _z_ _j_ +l = . . . = _z n_ = 0. Therefore,

Now (15) shows that the first _j_ − 1 columns of _A j_ are not changed in the multiplication _U jAj_.

We also observe that, since _2ww_ * has zeros in rows 1, . . . , _j_ − 1, the first _j_ − 1 rows of _A j_ are, likewise, not changed in the multiplication _U jAj_.

To perform the multiplication _U jAj_, we do _not_ first form the matrix _U j_. If _x_ is the _k_ th column of  , then the _k_ th column of the product _U jAj_ is

In real Euclidian space, the matrix _I_ − 2 _ww_ * represents the easily-visualized geometric transformation in Figure 7.8. Through the origin there is a plane, _π_ , to which _w_ is a unit normal. The vector ( _w_ * _x_ ) _w_ is the projection of _x_ in the direction of _w_. If we subtract twice this projection from _x_ , we obtain the vector

**Figure 7.8**

which is the _reflection_ of _x_ through the plane _π_.

#### **PROBLEMS**

**1.** Let _α_ , _β_ , _γ_ , _δ_ be real. Let

Reduce _A_ to right-triangular form by a transformation ( _I_ − 2 _ww_ *) _A_ = _R_.

**2.** Using the method described in this section, reduce the matrix

to right-triangular form.

### **7.15 THE _QR_ METHOD FOR COMPUTING EIGENVALUES**

In this section we will present an introduction to the method of Francis and Kublanovskaya, known as the _QR_ method. We wish to compute the eigenvalues of an _n_ × _n_ real or complex matrix _A_. By the method of the preceding section, we factor _A_ in the form

where _Q_ 1 _Q_ 1* = _I_ and where _R_ 1 is right-triangular. Setting _A_ = _A_ l, we now form

We now factor _A_ 2 in the form _A_ 2 = _Q_ 2 _R_ 2, and we form _A_ 3 = _R_ 2 _Q_ 2. In general,

Under certain conditions on _A_ , _as_ s → ∞ _the matrix_ As _becomes a right-triangular matrix with the eigenvalues of_ A _on the diagonal_. We will prove this convergence in the simplest case; we shall assume that _A_ has eigenvalues with distinct moduli:

This proof and more general proofs have been given by Wilkinson.

First, we discuss the transition from _A s_ to _A_ _s_ +l. By the method of the preceding section, we can find unitary, Hermitian matrices _U_ 1, . . ., _U_ _n_ − 1, depending on _s_ , such that

Thus, _A s_ = _Q_ _s_ _R_ _s_ , where _Q s_ = _U_ 1 . . . _U_ _n_ − 1. Then

The matrix _Q_ _s_ is never computed explicitly, nor are the matrices _U j_. We know that each _U j_ has the form

as in formula (15) of the preceding section. We do compute the scalars and vectors _ζ_ _j_ and _q_ ( _j_ ), and by means of them we compute successively

Then we compute successively

Every product _X_ ( _I_ – ζ _qq_ *) occurring in (8) is computed as _X_ – ζ _yq_ *, where _y_ = _Xq_.

We shall now study the _uniqueness_ and the _continuity_ of _QR_ factorizations.

**Lemma 1.** _Let Abe a nonsingular matrix with two_ QR _factorizations_

_where the_ Q's _are unitary and the_ R's _are right-triangular. Then there is a diagonal matrix of the form_

_such that_

_Proof_. First suppose that _Q_ = _R_ , where

Then _I_ = _Q_ * _Q_ = _R_ * _R_. The first row of _R_ * _R_ is

_r_ 11( _r_ 11, _r_ 12, . . . , _r_ 1 _n_ )

Therefore,

Since _r_ 12 = 0, the second row of _R_ * _R_ is

Therefore,

Continuing in this way, we verify that _r ij_ = _δ ij_( _i_ , _j_ = 1, . . . , _n_ ). Therefore, **_Q_** = **_R_** = **_I_**.

If _A_ = _Q_ 1 _R_ 1 = _Q_ 2 _R_ 2, where _A_ is nonsingular, then _R_ 1−1 exists and

If the diagonal elements of the right-triangular matrix _R_ 2 _R_ 1−1 have arguments _θ_ l, . . . , _θ_ n, and if _E_ is defined by (10), then the matrices

satisfy the conditions (12), and _Q_ = _R_. Therefore, _Q_ = _R_ = _I_ , and the identities _Q_ 2 = _Q_ 1 _E_ *, _R_ 2 = _ER_ 1 follow.

**Lemma 2.** _Let_ Ak → I _as_ k → ∞. _Let_ Ak = QkRk, _where_ Qk _is unitary, and where_ Rk _is right-triangular with positive-diagonal elements. Then_ Qk → I _and_ Rk → I _as_ k → ∞.

_Proof_. We have

Let _r ij_( _k_ ) be the _i_ , _j_ component of _R_ _k_. The first row of   is

Hence,

The second row of   is

Using the first result (15), we conclude that

Continuing by successive rows in this way, we conclude that _R_ _k_ → _I_ as _k_ → ∞. The explicit form of the inverse  . (cofactor matrix) _T_ now shows that  , and hence also   as _k_ → ∞.

**Lemma 3.** _If_ Q1, Q2, . . . _are unitary, and if_ R1, R2, . . . _are right-triangular, and if_

_then, for_ s = 2, 3, . . . ,

_and_

_Proof_. The identities (17) imply that

Therefore, if (18) is true for any _s,_ we have

Since (18) is true for _s_ = 1, it is true for all _s_.

Formula (19) also follows by induction from (17), since

**Theorem 1.** _Let_ A1 _be an_ n × n _matrix with eigenvalues_ λj _such that_ | _λ_ 1| > | _λ_ 2| > · · · > | _λ_ _n_ | > 0. _Let_ A1 _have a canonical diagonalization_

_where_ D = diag ( _λ_ 1, . . . , _λ_ n). _Assume that all of the_ n _principal minors of the matrix_ Y = X−1 _are nonsingular:_

_Let matrices_ A2, A3, . . . _be formed by the_ QR _method_ (17). _Then there is a sequence of diagonal matrices_ E1, E2, . . . _of the form_ (10) _such that_

_where the matrix_ Rx _is right-triangular_.

In this sense, _A_ _s_ tends to a right-triangular matrix with the eigenvalues of _A_ , in order, appearing on the main diagonal.

_Proof_. As we proved in Section 7.3, the condition (21) is necessary and sufficient for the matrix _Y_ = _X_ −1 to have a decomposition _Y_ = _L_ _Y_ _R_ _Y_ , where _L_ _Y_ is left-triangular, where _R_ _Y_ is right-triangular, and where _L_ _Y_ has 1's on the main diagonal. Then

But

since the elements above the diagonal are zero, while the elements on the diagonal are 1's, and the elements below the diagonal satisfy

Let _X_ have the factorization _X_ = _Q_ _x_ _R_ _x_ where _Q_ _x_ is unitary and _R_ _x_ is right-triangular. Then

where

Since _J_ _s_ → _I_ as _s_ → ∞, _J_ _s_ has a _QR_ factorization

where   is unitary and   is right-triangular; that follows from Lemma 2. Then   has the _QR_ decomposition

But Lemma 3 gives the _QR_ decomposition

Now Lemma 1 states that there is a matrix _E_ _s_ of the form (10) such that

Now the identity (19) for _A_ _s_ yields

But

Therefore,

and

This is the justification of the _QR_ method in the simplest case. It is possible to remove the restriction that _X_ −1 have nonsingular principal minors. It is even possible to remove the restriction that the eigenvalues of _A_ have distinct moduli. If the moduli are not distinct, the matrices _A_ _s_ tend to block-right-triangular matrices, where each _m_ × _m_ block centered on the diagonal is an _m_ × _m_ matrix whose _m_ eigenvalues are eigenvalues of equal moduli belonging to _A_. The theory underlying this method is surprisingly difficult.

The idea for the _QR_ method came, no doubt, from an older method due to Rutishauser:

where _L_ 1, _L_ 2, . . . are left-triangular and _R_ 1, _R_ 2, . . . are right-triangular.

#### **PROBLEMS**

_In the following problems, assume that all matrices_ L _are left-triangular matrices with_ 1's _on the main diagonal, and that all matrices_ R _are right-triangular matrices_.

**1.** If A = _L_ 1 _R_ 1 = _L_ 2 _R_ 2, and if _A_ is nonsingular, prove that _L_ 1 = _L_ 2 and _R_ 1 = _R_ 2.

**2.** If _A_ _k_ = _L_ _k_ _R_ _k_ → _I_ as _k_ → ∞, prove that _L_ _k_ → _I_ and _R_ _k_ → _I_.

**3.** If _A_ 1 is a nonsingular matrix, and if

prove that

**4.** * Let _A_ 1 satisfy the conditions of Theorem 1. Assume that matrices _A_ 2, _A_ 3, . . . can be formed as in the last problem. Show that, as _s_ → ∞, _A_ s tends to a right-triangular matrix whose diagonal elements are the eigenvalues _A_ 1.

* * *

 J. H. Wilkinson, _Rounding Errors in Algebraic Processes_. Englewood Cliffs, N.J.: Prentice-Hall, Inc., 1963, pp. 86–91.

 J. H. Wilkinson, "Convergence of the _LR_ , _QR_ , and related algorithms," _The Computer Journal_ , vol. 8 (1965), 77–84.

## **INDEX**

Abstract algebra, ,

Adjoint matrices, defined, , ; _see also_ Hermitian matrices

Associativity, , ,

Bandwidths of matrices, 208–15

Bilinear form,

Binomial coefficient, asymptotic form for,

Biorthogonality, principle of,

Canonical diagonalization of Hermitian matrices,

Canonical forms:

diagonal, 75–79

Jordan, , 121–39

numerical instability of, 232–33

Cauchy-Schwarz inequality, 84–85,

Cayley-Hamilton theorem, 113–14, 127–31

Characteristic equation, , 71–72

Column expansion, 10–13

Column vectors, defined,

Commutativity, 15–16, ,

Companion matrix,

Complex numbers, as field, 27–30,

Condition-number of a matrix, 174–77

defined,

Conservation of energy, law of, 104–5

Convex bodies, vector norms and, 171–73

Courant minimax theorem, 146–18

Cramer's rule, ,

Derivative of a determinant, 23–24

Determinants, 1–24

defined, 3–4

derivatives of, 23–24

evaluation of rank by, 43–47

of matrix products, 20–23

properties of, 5–9

Vandermonde's, 12–13

as volumes, 153–57

Diagonal canonical form, 75–79

Diagonalization:

canonical, of Hermitian matrices,

simultaneous, of two quadratic forms, 103–8

Differential equations, matrix analysis of, 57–70

Dimension of a linear space, 33–37

defined,

rank as,

Disjoint components of Gershgorin circles,

Distributivity, ,

Eigensolutions, 67–70

Eigenvalues:

basis for theory of,

computation of eigenvectors from, 228–31

defined,

dominant:

and computation of smaller eigenvalues, 241–50

defined,

method of iteration for, 235–39

in left-hand plane, 268–78

matrices with distinct, 71–74

multiplicities of, , 76–77

_QR_ method of computing, , 283–89

of similar matrices, 76–77

solution of differential equations by, 66–70

trace as sum of, 80–82

of tridiagonal and Hessenberg matrices, 251–56

variational properties of, 141–92

continuous dependence on matrices, 191–92

Eigenvectors:

basis for theory of,

computation of, from known eigenvalues, 228–31

defined,

principal axes of ellipsoids as, 95–98

solution of differential equations by, 66–70

of tridiagonal and Hessenberg matrices, 251–56

variational properties of, 141–92

Elimination, method of, 195–202

Ellipsoids, principal axes of, 94–98

Energy, law of conservation of, 104–5

Equal matrices, 14–15

Errors:

in computed eigenvalues and eigenvectors, 141–92

rounding, reduction of, 201–2, 217–21

Euclidean space,

rigid motions in, 117–20

Even permutation, defined,

Expansions, row and column, 10–13

Exponential matrix, solution of differential equations by, 64–65

Factorization by triangular matrices, 203–8

Fibonacci numbers,

Field, defined, 26–27

Finite-dimensional linear vector space,

basis of, 89–93

Francis and Kublanovskaya, method of ( _QR_ method), , 283–89

Fredholm theory of integral equations,

Functional analysis,

Gauss-Seidei method, 221–27

Generalized unit sphere, defined,

Gershgorin's theorem, 161–63

Sturm's theorem with, 255–56

Gram-Schmidt orthogonalization process, 89–93,

Guillemin, E. A.,

Hadamard's inequality, 153–57

Hermitian matrices, 98–102,

commuting, 115–17

factorization and, 206–8

Hessenberg, eigenvalues and eigenvectors of, 251–58

positive-definite, 105–8

determinant criterion for, 152–53

Gauss-Seidel method and, 224–27

Rayleigh principle and, 141–45

Weyl's theorem and, 157–61

Hessenberg matrices, eigenvalues and eigenvectors of, 251–58

Homogeneous systems:

defined,

solvability of, 38–42

Householder and Bauer, method of, , 258–67

Hurwitz,

Inclusion principle, 149–51

as determinant criterion for positive definiteness, 152–53

Inequalities:

Cauchy-Schwarz, 84–85,

Hadamard's, 153–57

Weyl's, 157–61

Inhomogeneous systems:

defined,

fundamental solution of,

general _m_ × _n_ , 48–49

reduction of, by matrix analysis, 60–62

Inner products, defined, , ,

Invariants, 80–82

Inverse matrix, 17–19

Irreducible matrices, 181–84

Iterative methods, 221–27

for dominant eigenvalues, 235–39

Jordan canonical form, , 121–39

numerical instability of, 232–33

Jordan's theorem, 122–25,

proof of, 132–39

Keller's theorem, 224–27

Kronecker delta,

Laguerre polynomials,

Least-squares solution, 50–55

Linear combinations, 33–37

Linear equations:

differential, matrix analysis of, 57–70

direct solution of large systems of, 208–15

theory of, 26–55

Linear space, _see_ Linear vector space

Linear vector space, 26–32

basis of, 33–37

of mutually orthogonal unit vectors, 89–93

dimension of, 33–37

rank as dimension,

finite-dimensional,

basis of, 89–93

Linearly dependent and independent vectors, 33–37

Lyapunov's theorem, 270–72

Mass-spring systems, 103–8

Matrix, defined,

Matrix norms,

related, 167–71

defined, 167–68

Matrix products, determinants of, 20–23

Metric, Riemannian,

Minimax theorem, Courant, 146–48

Multiplication of matrices, 15–16,

Newton's method,

Nonsingular matrix, defined,

Normal equations, 51–55

defined,

Normal matrices, 115–20

defined,

Norms, vector, 163–73

convex bodies and, 171–73

regular, defined,

Null space, ,

defined,

Numbers as fields, 26–32

Numerical instability of the Jordan canonical form, 232–33

Numerical methods, 194–289

computation of eigenvectors, 228–31

direct solution of large systems of linear equations, 208–15

elimination, 195–202

factorization by triangular matrices, 203–8

identification of stable matrices, 268–78

iterative methods, 221–27

for dominant eigenvalues, 235–39

to obtain the smaller eigenvalues, 241–50

method of Householder and Bauer, , 258–67

_QR_ method, , 283–89

reduction of rounding error, 201–2, 217–21

with tridiagonal and Hessenberg matrices, 251–58

Odd permutation, defined,

Orthogonality:

defined,

Gram-Schmidt process for, 89–93

of principal axes of ellipsoids, 95–98

Parallelograms, Hadamard's inequality and, 154–56

Permutation:

even, defined,

odd, defined,

sign of the,

theorems on reversal of, ,

Perron's theorem, 177–82

Perturbation theory, 141–92

described,

Pivotal element, 201–2

Positive and irreducible matrices, 177–84

Positive-definite matrices, 105–8

determinant criterion for, 152–53

Gauss-Seidel method and, 224–27

Principal axes of ellipsoids, 94–98

Principal vectors, 125–31

defined, 125–26

_QR_ method, , 283–89

Rank of a matrix,

defined, 40–41

evaluation of, by determinants, 43–47

Rational numbers, as field,

Rayleigh principle, 141–45,

Rayleigh quotient, ,

Real numbers, as field, , , , ,

Related matrix-norms, 167–71

defined, 167–68

Riemannian metric,

Rigid motions in Euclidian space, 117–20

Rotation, theory of normal matrices and, 117–20

Rounding error:

in method of iteration for dominant eigenvalues,

reduction of, 201–2, 217–21

Routh's theorem, 268–70

Row expansion, 10–13

Row vectors, defined,

Rutishauser's method,

Scalars, defined,

Schwarz inequality, _see_ Cauchy-Schwarz inequality

Sign of the permutation,

theorems on reversal of, ,

Similar matrices:

defined,

eigenvalues of, 76–77

Singular matrix, defined,

Skew-Hermitian matrices, 115–17

Space, _see_ Euclidean space _and_ Null space

Spectrum, perturbations of, 186–89

Stable matrices, numerical identification of, 268–78

Sturm's theorem, 254–56

Submatrices:

defined,

evaluation of rank and, 43–47

Successive over-relaxation,

Symmetric matrices, 94–95,

mass and spring matrices as,

Trace, 80–82

defined,

Transpose of a matrix, formation of,

Triangular matrices:

accurate unitary reduction to, 279–83

factorization by, 203–8

recursion solution of,

Triangularization, unitary, 109–14

Tridiagonal matrices, eigenvalues and eigenvectors of, 251–58

Uncoupling property of canonical diagonal form, 78–79

Unit vectors:

defined,

mutually orthogonal, 89–93

Unitary matrices, 83–88,

accurate reduction of, to triangular form, 279–83

defined,

as normal matrices, 115–17

Unitary triangularization, 109–14

Unsolvablc systems, least-squares solution of, 50–55

Vandermonde's determinant, 12–13

Varga, R. S.,

Variational principles, 141–92

Vector addition, 27–28

Vector norms, 163–73

convex bodies and, 171–73

regular, defined,

Vector space, _see_ Linear vector space

Vectors, _see also_ Eigenvectors

defined,

length of, and unitary matrices, 83–87

linearly dependent and independent, 33–37

matrices and, 14–16

principal, 125–31

defined, 125–26

unit:

defined,

mutually orthogonal, 89–93

Volumes, determinants as, 153–57

Weyl's inequalities, 157–61

Wilkinson, J. II., , ,

label: mathematics

title: Matrix Theory

authors: Joel N. Franklin