## Understanding Ordinary Least Squares (OLS)

The Ordinary Least Squares (OLS) method is a statistical method for estimating the parameters of a linear regression model.

It is one of the most commonly used methods in data science and machine learning. OLS is based on the concept of minimizing the sum of squared errors between observed values and predicted values. The derivation of OLS involves taking partial derivatives with respect to each parameter, setting them equal to zero, and solving for the parameters in terms of other variables. This process can be used to find an optimal solution for any linear regression problem.

By using this method, we can find an optimal solution that minimizes the sum of squared errors between observed values and predicted values. In the context of linear regression, OLS is used to estimate the parameters of a model. In this case, the model is a line that represents an observed point in time and its corresponding predicted value for a selected variable.

If Æ’ is any function relating x and y, then Æ’(x) = Î± + Î²x: Æ’(y) = Î± + Î²y. The term “ordinary least squares” comes from the fact that it minimizes squared errors taken as observations minus predictions.

## Ordinary Least Square (OLS) Derivation

Consider a linear regression model with one independent variable: Y_{i} = Î±̂ + Î²̂X_{i} + e_{i} where Y_{i} is the response variable, X_{i} is the independent variable, alpha hat (Î±̂) and beta hat (Î²̂) are the coefficients, and e_{i} is the error term.

The model is given by:

Y_{i} = Î±̂ + Î²̂X_{i} + e_{i}

_{}

Where Y_{i} is the response variable, X_{i} is the independent variable, Î±̂ and Î²̂ are the coefficients to be estimated, and e_{i} is the error term.

To estimate Î±̂ and Î²̂ using the OLS method, we want to minimize the sum of squared errors (SSE) between the predicted values Å¶_{i} and the actual values Y_{i}:

Where n is the number of observations.

The predicted values Å¶_{i} are given by:

Å¶_{i} = Î±̂ + Î²̂X_{i}

To find the values of Î±̂ and Î²̂ that minimize SSE, we take the partial derivatives of SSE with respect to Î±̂ and Î²̂ and set them equal to zero:

∂SSE/∂Î±̂ = -2 (Y_{i}- Î±̂ - Î²̂X_{i}) = 0∂SSE/∂Î²̂ = -2XX_{i}(Y_{i}- Î±̂ - Î²̂_{i}) = 0

Solving these equations simultaneously, we obtain the following OLS estimators for Î±̂ and Î²̂:

Î±̂ = È² – Î²̂X̄

Î²̂ = (X_{i} – X̄)×(Y_{i} – È²) / (X_{i} – X̄)^{2}

^{}

where X̄ and È² are the sample means of X and Y, respectively.

Thus, the OLS estimates for Î±̂ and Î²̂ are given by:

Î±̂ = È² – Î²̂X̄

Î²̂ = (X_{i} – X̄)*(Y_{i} – È²) / (X_{i} – X̄)^{2}

^{}

where È² is the sample mean of Y, X̄ is the sample mean of X, and n is the number of observations.

Once we have estimated Î±̂ and Î²̂, we can use them to make predictions for new values of X by plugging them into the equation:

Å¶ = Î±̂ + Î²̂X

### Properties of least square estimators using Gauss-Markov Theorem

The Gauss-Markov theorem provides a set of conditions under which the Ordinary Least Squares (OLS) estimator is the Best Linear Unbiased Estimator (BLUE). The properties of the OLS estimator can be summarized as follows:

**Unbiasedness:**The OLS estimator is unbiased, which means that it has an expected value that is equal to the true population parameter. In other words, on average, the OLS estimator produces estimates that are close to the true values of the parameters being estimated.**Efficiency:**Among all linear unbiased estimators, the OLS estimator has the smallest variance. This means that it produces estimates that are more precise than any other linear unbiased estimator and achieves the smallest possible mean squared error.**Consistency:**As the sample size n approaches infinity, the OLS estimator approaches the true population parameter with probability 1. In other words, as the sample size becomes larger, the OLS estimator becomes more and more accurate.**Normality:**Under certain assumptions, the OLS estimator is normally distributed. This is useful for constructing confidence intervals and hypothesis tests for estimated parameters.

The Gauss-Markov theorem provides conditions under which the OLS estimator satisfies these properties. The conditions include:

**Linearity:**The regression model is linear in the parameters.**Strict exogeneity:**The error term has a zero mean and is uncorrelated with the independent variables.**No perfect multicollinearity:**There is no perfect linear relationship among the independent variables.**Homoscedasticity:**The error term has constant variance.**Normality:**The error term is normally distributed.**Sample size:**The sample size is sufficiently large.

If these conditions are met, then the OLS estimator is the Best Linear Unbiased Estimator (BLUE), meaning it has the smallest variance among all unbiased linear estimators. In other words, the OLS estimator is the most efficient and precise estimator of the population parameters in a linear regression model.

**Suggested Articles:**

**Econometrics & its Origin and Definitions****Nature, Types, and Sources of Econometric Data****Procedures of Econometric Modelling****Uses and Applications of Econometrics****Relationships between Econometrics, Mathematics, and Statistics****Business Econometrics – Importance and Procedures****Environmental econometrics – Meaning, Uses, and Procedures****Estimation and Types of Estimation****Properties of Estimators – Small Sample and Large Sample****Ordinary Least Squares (OLS) Derivation**

Please leave your comments or ask your queries here. The comments shall be published only after the Admin approval.