Logistic Regression Explained: From Sigmoid to a Working Classifier (with Python)

What you'll learn

Logistic regression is the classification counterpart to linear regression and the single most useful baseline for any yes/no problem. It is fast, interpretable, and calibrated — it outputs real probabilities, not just labels.

By the end you will be able to:

• Explain why a 'regression' algorithm is used for classification
• Understand the sigmoid function and what its output means
• Know why log-loss, not MSE, is the right loss
• Tune the decision threshold and handle class imbalance
• Build and evaluate a classifier with precision, recall, and ROC-AUC

Why 'regression' when it classifies?

Logistic regression starts exactly like linear regression — it computes a weighted sum of the inputs, z = w·x + b. The twist is what happens next: instead of using z directly, it squashes it through the sigmoid function to turn it into a probability between 0 and 1.

The name is historical: the model is linear in the log-odds of the outcome. So it really is a regression — just on the log-odds rather than the value itself.

The sigmoid function

The sigmoid (logistic) function maps any real number to the range (0, 1):

sigmoid(z) = 1 / (1 + e^(-z))

A large positive z gives an output near 1, a large negative z gives near 0, and z = 0 gives exactly 0.5. That output is interpreted as the probability that the example belongs to the positive class. You then apply a threshold (usually 0.5) to get a hard label.

The loss function: log-loss

For classification we do not use MSE. Instead we use log-loss (binary cross-entropy):

log_loss = -(1/N) * Σ [ y*log(p) + (1-y)*log(1-p) ]

Intuitively it punishes confident wrong answers severely: predicting 0.99 for something that was actually class 0 incurs a huge penalty. Two reasons we prefer it over MSE here: with the sigmoid, MSE is non-convex (full of local minima that trap gradient descent), whereas log-loss is convex and trains reliably; and log-loss produces stronger, cleaner gradients.

Interpreting the coefficients

Each coefficient tells you how a feature shifts the log-odds. Exponentiating it gives the odds ratio: exp(w) is the multiplicative change in the odds of the positive class for a one-unit increase in that feature, holding the others fixed. This interpretability is why logistic regression is still trusted in medicine, credit scoring, and any regulated setting.

Implementing it in Python

We will classify tumours as malignant or benign using the built-in breast-cancer dataset:

from sklearn.linear_model import LogisticRegression
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import accuracy_score, roc_auc_score

X, y = load_breast_cancer(return_X_y=True)
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.2, random_state=42)

# Scaling helps the solver converge and keeps regularisation fair.
scaler = StandardScaler().fit(Xtr)
Xtr, Xte = scaler.transform(Xtr), scaler.transform(Xte)

clf = LogisticRegression(max_iter=1000)
clf.fit(Xtr, ytr)

proba = clf.predict_proba(Xte)[:, 1]
print("accuracy:", round(accuracy_score(yte, clf.predict(Xte)), 4))
print("ROC-AUC :", round(roc_auc_score(yte, proba), 4))

This prints something like:

accuracy: 0.9737
ROC-AUC : 0.9974

An ROC-AUC of 0.997 means the model almost always ranks a random positive above a random negative. Note the StandardScaler — logistic regression's solver converges faster and its regularisation is fairer when features share a scale.

The decision threshold

The default 0.5 cut-off is rarely optimal. Lowering it catches more positives (higher recall) at the cost of more false alarms (lower precision); raising it does the opposite. Choose the threshold from the business cost of each error type:

• Cancer screening: missing a positive is catastrophic — lower the threshold to maximise recall.
• Spam filter: deleting a real email is worse than one spam getting through — raise the threshold to protect precision.

Handling class imbalance

If only 1% of examples are positive, a model that always predicts 'negative' is 99% accurate and useless. Fixes:

• Use class_weight='balanced' so the loss weights the rare class more heavily.
• Resample: oversample the minority (e.g. SMOTE) or undersample the majority.
• Stop trusting accuracy — evaluate with precision, recall, F1, and PR-AUC instead.

Multi-class classification

For more than two classes, logistic regression extends in two ways: one-vs-rest trains one binary classifier per class, while softmax (multinomial) regression generalises the sigmoid to output a probability distribution over all classes at once. scikit-learn picks a sensible default automatically.

Common pitfalls

• Reporting accuracy on imbalanced data. It hides the failure on the class you actually care about.
• Forgetting to scale features when using regularisation.
• Assuming a non-linear boundary. The decision boundary is linear unless you add polynomial or interaction features.
• Leaving the threshold at 0.5 without checking the precision–recall trade-off.

Key takeaways

• Logistic regression = linear score → sigmoid → probability → threshold.
• It is trained with convex log-loss and outputs interpretable, calibrated probabilities.
• Tune the threshold and use class weights for imbalance; judge with precision/recall and ROC-AUC.
• It is the classification baseline every project should beat before reaching for something fancier.

Further reading: scikit-learn docs, StatQuest's Logistic Regression series, and the visual guide.