..

Rosetta Stone of Linear Algebra

While learning Linear Algebra, I’ve found that, very often, I go back and revisit certain concepts, and try to geometrically understand what certain concepts represent. I found myself engaging in this excercise quite often, and it helped me deepen my intuition for the subject. Often times, a single explanation for a concept may not be sufficient to fully capture the oddities of mathematical object like Matrices. It’s necessary to ponder over it, and turn the object around from various different angles to get “it”. I’ve found “it” is as elusive as ever, and end up learning something that I quite didn’t understand each time I revisit something I thought I already knew well enough. These are some of the questions I found myself asking over and over again (some not as much as others).


  1. What does the dot product of two vectors represent?
  2. What is the euclidean norm of a vector?
  3. What is the Cauchy Schawrz Inequality?
  4. Can you see that matrices are representations of Linear Transformation in a n-dimensional space?
  5. What do rectangular matrices represent?
  6. What do the basis vectors represent in a n-dimensional space?
  7. What is meant by a Rank of a Matrix?
  8. What is meant by a Singular matrix?
  9. What is meant by a Span of a Matrix?
  10. What do we mean when we say, a set of vectors are linearly independent? Or, when is a set of vectors linearly independent/dependent?
  11. What is a subspace? How to think about Rank in the context of Subspaces?
  12. What does the four fundamental subspaces, i.e, Column Space, Row Space, Left-Null Space, Null Space/Kernel represent?
  13. What does the determinant of a matrix represent?
  14. Why is the determintant for a matrix with linearly dependent set of column vectors zero?
  15. What does the inverse of a matrix represent?
  16. What does a Pseudoinverse of a rectangular matrix represent?
  17. What strategies can you think of when in Ax = y (A not necessarily a square matrix), where y is not in the column space of A?
  18. What does AB represent where A and B are both matrices in a n-dimensional space?
  19. What do peudoinverses represent and in which scenarios can they be useful?
  20. Do you understand how Projections works? Think of Projecting a vector onto another vector. And then think of projecting a vector onto a m-dimensional subspace spanned by some set of column vectors in, say, A.
  21. Do you understand how to decompose a vector into two orthogonal vectors?
  22. Can you visualize QQ^t where Q is an orthogonal matrix? And can you see how Q.Q^t is I?
  23. Can you see how Orthogonal Projections to a m dimensional subspace of R^n space of a nxm matrix (n > m) would function like?
  24. What kind of linear transformation effect does a diagonal/orthogonal/symmetric/skew-symmetric etc have on a vector space as a whole?
  25. How are Orthogonal projections used for least-squares model fitting?
  26. What do eigen values and eigen vectors represent?
  27. Can you see why Eigen value decomposition (Or Spectral Decomposition) works?
  28. Can you reason why Eigenvectors are sign ambiguous?
  29. Can you reason why Eigenvectors of symmetric matrices are pair-wise orthogonal?
  30. What do singular values in the context of SVD Represent?
  31. Can you see how the original matrix can be arrived from spectral decomposition through repeated addition of rank-1 approximations?
  32. Can you geometrically understand SVD for any NxM matrices?
  33. Can you visualize what Quadratic forms represent?
  34. What are positive semi-definite (PSD) matrices? And what are the different types of Definiteness of Matrices and why do mean in relation to Quadratic forms?


The identities that follow from each of these questions is not of importance in this article, as it can be understood by extension of having understood the concepts the questions try to focus on. In most cases, understanding the underlying intuition behind these concepts should be enough to understand why certain identities hold true without the need for outlining a formal proof. Though it may not be applicable in all of the cases, it should hold true for most of the cases. Which should give you sufficient motivation to then go ahead and understand those tricky parts later. I’ve found a great resource which outlines many such identities (but not all of them) here

I will be adding to this list of questions as I study more of the subject matter, and am introduced to more not-so-obvious aspects of the subject.