Chapter 2 The Geometry of Linear Regression Introduction In Chapter 1, we introduced regression models, both linear and nonlinear, and discussed how to estimate linear regression models by using the method of moments. | Chapter 2 The Geometry of Linear Regression Introduction In Chapter 1 we introduced regression models both linear and nonlinear and discussed how to estimate linear regression models by using the method of moments. We saw that all n observations of a linear regression model with k regressors can be written as y X3 u 2-01 where y and u are n-vectors X is an n x k matrix one column of which may be a constant term and 3 is a k-vector. We also saw that the MM estimates usually called the ordinary least squares or OLS estimates of the vector 3 are 3 XTX -1XTy- 2-02 In this chapter we will be concerned with the numerical properties of these OLS estimates. We refer to certain properties of estimates as numerical if they have nothing to do with how the data were actually generated. Such properties hold for every set of data by virtue of the way in which 3 is computed and the fact that they hold can always be verified by direct calculation. In contrast the statistical properties of OLS estimates which will be discussed in Chapter 3 necessarily depend on unverifiable assumptions about how the data were generated and they can never be verified for any actual data set. In order to understand the numerical properties of OLS estimates it is useful to look at them from the perspective of Euclidean geometry. This geometrical interpretation is remarkably simple. Essentially it involves using Pythagoras Theorem and a little bit of high-school trigonometry in the context of finite-dimensional vector spaces. Although this approach is simple it is very powerful. Once one has a thorough grasp of the geometry involved in ordinary least squares one can often save oneself many tedious lines of algebra by a simple geometrical argument. We will encounter many examples of this throughout the book. In the next section we review some relatively elementary material on the geometry of vector spaces and Pythagoras Theorem. In Section we then discuss the most important numerical properties
