This summer for fun I did the first four courses of Harvard EdX Ph525 The part I was really trying to dig into was the math behind Factor Analysis, which they ended up barely scratching the surface of.

So, I pretty much understand the basics of singular value decomposition, and how that relates to Principal Components. I am trying to help a friend do some custom work involving "Confirmatory Factor Analysis", involving restrictions (e.g., restricting the factor loadings for two sets of data to be equal, and testing the LaGrange multipliers/scores to see how much relaxing that restriction would improve the fit. (I am aware of and will be using the lavaan package in R- but need to understand exactly what is going on in the background.) Any suggestions of a good book or other resource that goes through the guts of the mathematics behind all of this?

I am decent enough at math/stats so that as long as the language they are speaking is in terms of matrix algebra and/or linear models, I'll be able to get it. But, if there is a book that combines the theory with a little bit of applied context, that would be wonderful as well. Thank you in advance for your time!