projection matrix

If \(A\) has linearly independent columns, then \((A^TA)^{-1}A^T b\) gives the vector \(c\) such that \(Ac\) is the projection of \(b\) onto the column space of \(A\). Or: "I want to find the projection of \(b\) onto \(col(A)\). Give me the coordinates c in the column space \(A\) of this projection.":

• Consider the vector \(b\) you are trying to project.
• Let \(b_A\) be the projection of \(b\) onto \(col(A)\). Then, \((b - b_A)\) is orthogonal to the column space of \(A\). That is, it is in the nullspace of \(A^T\), because the dot product with every column vector of \(A\) is 0.
• That is, \(A^T(b-b_A) = 0\)
• Since \(b_A\) is in the column space of \(A\), there is some \(c\) such that \(b_A = Ac\).
• So \(A^T(b - Ac) = 0 \Rightarrow A^Tb = A^TAc \Rightarrow (A^TA)^{-1}A^Tb = c\)
• Note: if the columns of \(A\) are not orthogonal, the coordinates are not unique

• textbook
• 18.06 notes