Linear Regression for Machine Learning: The Complete Beginner's Guide (with Python)

What you'll learn

Linear regression is the first algorithm almost everyone learns, and for good reason: it is simple enough to work out by hand, yet it introduces nearly every idea you will reuse in more complex models — a loss function, an optimisation method, overfitting, and regularisation.

By the end of this guide you will be able to:

• Explain what linear regression is and where it fits in machine learning
• Write down the model and read its coefficients
• Understand the two main ways it is trained — the normal equation and gradient descent
• Work through a tiny example by hand
• Fit, evaluate, and interpret a model in a few lines of scikit-learn
• Prepare your data properly and avoid the classic mistakes

You do not need heavy maths. If you are comfortable with the equation of a straight line, y = mx + c, you already know the core idea.

Isn't linear regression just statistics?

Yes — and that is fine. Linear regression predates machine learning by more than a century, and statisticians use it to understand relationships between variables. Machine learning borrows the same model but shifts the goal: instead of explaining the past, we want to predict new, unseen values as accurately as possible.

The maths is identical. What changes is emphasis: an ML practitioner cares about generalisation (how well the model does on data it has never seen), uses train/test splits and cross-validation, and reaches for regularisation to control overfitting. Same equation, different mindset.

The intuition in plain English

Imagine plotting house size against price. Each house is a dot. Linear regression draws the single straight line that passes as close as possible to all the dots at once. Once you have that line, predicting the price of a new house is easy: find its size on the x-axis and read the price off the line.

With more than one input — say size, number of bedrooms, and distance to the station — the 'line' becomes a flat plane (or a higher-dimensional equivalent), but the idea is unchanged: find the flat surface that best fits the data.

The representation: what the model actually is

For a single input feature, the model is just the equation of a line:

y = w * x + b

Here w is the weight (the slope — how much y changes when x increases by one) and b is the bias (the intercept — the value of y when x is zero). With multiple features x₁, x₂, …, xₙ it generalises to a weighted sum:

y = w1*x1 + w2*x2 + ... + wn*xn + b

Training is simply the process of finding the best values of the weights and bias. 'Best' means the values that make the predictions closest to the actual answers.

How the model learns: the loss function

To find the best line we need to measure how wrong a given line is. The standard measure is Mean Squared Error (MSE) — the average of the squared gaps between what the model predicted and the true value:

MSE = (1/N) * sum( (y_actual - y_predicted)^2 )

We square the errors for three reasons: negative and positive errors do not cancel out, large errors are penalised more heavily, and the result is smooth and easy to differentiate. Training means finding the weights that make MSE as small as possible. There are two classic ways to do it.

1. The Normal Equation (closed form). There is an exact formula that solves for the best weights in one step:

w = (Xᵀ X)⁻¹ Xᵀ y

It is elegant and needs no tuning, but it inverts a matrix — roughly an O(n³) operation — so it becomes slow when you have very many features, and it fails when features are perfectly correlated.

2. Gradient Descent (iterative). Start with random weights, compute the slope (gradient) of the error with respect to each weight, and take a small step 'downhill'. Repeat until the error stops improving:

w := w - learning_rate * gradient_of_MSE_wrt_w

Gradient descent scales to millions of features and rows and is the workhorse behind neural networks. The learning rate controls the step size — too big and it overshoots, too small and it crawls.

A worked example by hand

Take five tiny data points and fit y = w·x + b:

Using the least-squares formulas, the slope is the covariance of x and y divided by the variance of x:

mean_x = 3, mean_y = 4
w = Σ(x-mean_x)(y-mean_y) / Σ(x-mean_x)²
  = (( -2*-2)+(-1*0)+(0*1)+(1*0)+(2*1)) / (4+1+0+1+4)
  = 6 / 10 = 0.6
b = mean_y - w*mean_x = 4 - 0.6*3 = 2.2

So the best-fit line is y = 0.6x + 2.2. Predicting for x = 6 gives 0.6·6 + 2.2 = 5.8. That is the entire algorithm — everything else is scaling this up.

Implementing it in Python

In practice you never code the formulas by hand — scikit-learn does it for you. Here we predict house price from area:

import numpy as np
from sklearn.linear_model import LinearRegression

# Area (sq ft) -> Price (in lakhs). One feature to keep it visual.
X = np.array([500, 750, 1000, 1250, 1500, 1750, 2000]).reshape(-1, 1)
y = np.array([25, 34, 46, 52, 63, 72, 84])

model = LinearRegression()
model.fit(X, y)

print("slope (w):", round(model.coef_[0], 4))
print("intercept (b):", round(model.intercept_, 4))
print("R^2:", round(model.score(X, y), 4))
print("predict 1600 sq ft:", round(model.predict([[1600]])[0], 2), "lakhs")

Running this prints:

slope (w): 0.0386
intercept (b): 5.5
R^2: 0.9962
predict 1600 sq ft: 67.21 lakhs

The R² of 0.9962 means the model explains about 99.6% of the variation in price — expected here because the data is almost perfectly linear. Plotting the fit makes it concrete:

import matplotlib.pyplot as plt

plt.scatter(X, y, color="teal", label="actual")
plt.plot(X, model.predict(X), color="crimson", label="best-fit line")
plt.xlabel("Area (sq ft)"); plt.ylabel("Price (lakhs)")
plt.legend(); plt.title("Linear Regression fit")
plt.show()

You will see the teal dots (actual prices) hugging the crimson best-fit line.

Preparing your data for linear regression

The model makes assumptions. Respecting them is what separates a good fit from a misleading one:

• Linearity. The relationship should be roughly straight. If it curves, add polynomial features or transform the variable (e.g. log).
• Remove or handle outliers. Because errors are squared, a single extreme point can drag the whole line toward it.
• Scale your features when using gradient descent or regularisation, so no single feature dominates just because of its units.
• Check for multicollinearity. Highly correlated inputs make the coefficients unstable; detect it with the Variance Inflation Factor (VIF) and drop or combine features.
• Gaussian-ish, constant-variance errors. Residuals should look like random noise with no pattern or fan shape when plotted against predictions.

Regularisation: Ridge and Lasso

When you have many features, plain linear regression can overfit — fitting noise instead of signal. Regularisation adds a penalty on large weights:

• Ridge (L2) shrinks all weights toward zero smoothly. Good when many features each contribute a little.
• Lasso (L1) can drive some weights exactly to zero, effectively performing feature selection. Good when you suspect only a few features matter.
• Elastic Net blends both.

In scikit-learn these are drop-in replacements: Ridge(alpha=1.0) or Lasso(alpha=0.1) in place of LinearRegression().

Common pitfalls

• Judging fit by R² alone. R² never drops when you add features — use adjusted R² or a held-out test set.
• Forgetting the train/test split. A model that looks perfect on training data may be memorising it. Always evaluate on unseen data.
• Using it for classification. For yes/no problems, use logistic regression — linear regression outputs unbounded numbers and is sensitive to outliers.
• Ignoring residual plots. A curved or fanning residual plot is the clearest sign an assumption is violated.

Key takeaways

• Linear regression fits the straight line (or plane) that minimises mean squared error.
• It is trained either by the normal equation (exact, small problems) or gradient descent (scalable).
• Ridge and Lasso regularisation control overfitting and can select features.
• It is the foundation for understanding loss functions, optimisation, and the bias-variance trade-off across all of ML.

Further reading: the scikit-learn linear models guide, StatQuest's Linear Regression video, and the interactive breakdown on our ML Algorithms visual guide.