I've picked up a few Stats and Physics books that I just can't read, and it's because the Math is hard. Hard in a weird way, to me anyway. Knowing quite a good bit of Math it's strange to me that I'd pick up a book in another field and not understand the Math in it. In the first chapter of an Econometrics book, casual appeal to a matrix form of Taylor's Theorem? Matrix operations defined by the Dirac delta function when that never comes up as a significant topic in a Linear Algebra book.
Well I think I need to just push harder on the Math. I have fairly short patience tracing through the less mathematically rigorous material that is standard in these other subjects, used to build up intuition. So I figure, more Math is always better and at the end of the tunnel I have a better chance of circling back to the other topics.
So these days I've been working on Measure Theory and Probability Theory, in order to get me to that great Bayesian mountain: Jaynes' Probability Theory. With Math muscles that big I'm sure most Econometric books should be no problem. I should also pick up some advanced Linear Algebra books or maybe something on the Math of Physics, and eventually be ready to learn relativity.