Logistic Regression: From Scores to Decisions

01 / THE MODELLING QUESTION

Why does a classifier have “regression” in its name?

Imagine predicting whether a customer will cancel a subscription in the next month. The observed outcome is binary: cancellation is 1, retention is 0. But before that month ends, a model should express uncertainty. A probability of 0.72 is more informative than a bare “yes”: it can support prioritisation, capacity planning and decisions with different error costs.

Binary logistic regression models the conditional probability p(x) = P(Y = 1 | X = x). It first forms a linear score from the features, then maps that score to a probability. A separate threshold turns the probability into a class label. These are three different operations.

Here x is a feature vector, w contains learned coefficients, b is an intercept, and t is a decision threshold. The score z can be any real number. The sigmoid produces a number strictly between 0 and 1 for finite z. That makes it a valid probability parameter, although validity alone does not establish that the probabilities are accurate.

Ordinary least squares applied directly to binary outcomes can predict values below zero or above one. Logistic regression instead specifies a Bernoulli outcome model with a linear relationship on the log-odds scale. Its probability curve is nonlinear, but the model remains linear in the supplied features on that transformed scale. The classic statistical treatment includes Cox’s work on binary sequences [1].

02 / INTERACTIVE EXPERIMENT

Turn a score into a probability

The sigmoid has a midpoint at z = 0, where the probability is exactly 0.5. Positive scores favour class 1; negative scores favour class 0. Equal movements in z do not produce equal movements in probability.

What the slope tells us

dσ/dz = σ(z)[1 − σ(z)] = p(1 − p)

The slope peaks at 0.25 when p = 0.5 and becomes small in the tails. Around an uncertain prediction, a small score change can noticeably alter the probability. Near an extreme prediction, the same score change has less effect. The symmetry σ(−z) = 1 − σ(z) explains why scores of equal magnitude on opposite sides of zero produce complementary probabilities.

Do not interpret z = 2 as “twice as likely” as z = 1. Scores are log-odds, and the transformation is nonlinear. The explorer reports the score, probability, odds and local slope together so that you can see these distinctions directly.

03 / INTERPRETATION & GEOMETRY

The linear relationship is in the log-odds

Odds compare the probability of an event with the probability of its complement. If p = 0.8, the odds are 0.8/0.2 = 4, often written 4:1. Taking their natural logarithm gives the logit. Rearranging the sigmoid yields:

Holding other features constant, increasing feature xⱼ by one unit changes the log-odds by wⱼ and multiplies the odds by exp(wⱼ). A coefficient of approximately 0.693 therefore doubles the odds. It does not double the probability: doubling odds from 1 to 2 moves p from 0.5 to about 0.667, whereas doubling odds from 4 to 8 moves p from 0.8 to about 0.889.

This interpretation is conditional on the fitted model, feature encoding and units. A coefficient on a standardised variable describes a one-standard-deviation change. Correlated predictors can make individual coefficients unstable. An association in a predictive model is not evidence that intervening on that feature would cause the predicted change.

Experiment: change the model, not the observation

Below, x is a hypothetical standardised feature. Adjust w and b to change the whole probability curve p(x) = σ(b + wx). The dashed horizontal line marks p = 0.5.

For w ≠ 0, the 0.5 crossing occurs at x = −b/w. A negative coefficient reverses the direction of the curve. With w = 0, x has no influence and all observations receive σ(b). In multiple dimensions, the threshold boundary is wᵀx + b = log[t/(1 − t)]. It is a hyperplane in the supplied feature space.

Adding x², interactions or spline features can create a curved boundary in the original input space. “Linear classifier” does not mean that feature engineering is forbidden; it identifies where the linearity assumption lives.

04 / FROM PROBABILITY TO LEARNING

Learning the coefficients with maximum likelihood

For one labelled observation, a Bernoulli model assigns probability pᵞ(1 − p)¹⁻ᵞ to the observed outcome. This compact expression selects p when y = 1 and 1 − p when y = 0. Under a conditionally independent-observations likelihood, multiplying these contributions gives the likelihood of the training labels.

Maximising likelihood is equivalent to minimising its negative logarithm. Dividing by n changes the scale, not the unregularised minimiser. The resulting objective is binary cross-entropy, also called log-loss. It evaluates how much probability the model assigned to the outcome that actually occurred.

Why the gradient is surprisingly simple

For one observation, differentiate the loss with respect to p and multiply by dp/dz = p(1 − p). The terms cancel, leaving ∂ℓ/∂z = p − y. Applying the chain rule to z = b + wᵀx gives:

If an observation has y = 1 but the model predicts a small p, its residual p − y is negative. The gradient update pushes its score upward, subject to the contributions of other observations. η is the learning rate. An overly large value can make updates unstable; a very small value can make progress slow.

The sigmoid’s small tail derivative does not by itself imply a tiny log-loss gradient for a confidently wrong example. For y = 1 and p close to zero, p − y is close to −1. This is a useful distinction between the derivative of the activation and the derivative of the complete objective.

Convexity is helpful—but not a guarantee of a finite solution

With the intercept included as a constant feature, the unregularised Hessian is XᵀWX/n, where W has diagonal entries pᵢ(1 − pᵢ). It is positive semidefinite, so the objective is convex in the coefficients. There are no worse local minima to escape. Yet rank deficiency can prevent uniqueness, and separable data can prevent a finite maximum-likelihood estimate altogether.

Libraries commonly use solvers such as L-BFGS or Newton-type methods rather than a hand-written fixed-step loop. Solver convergence and statistical generalisation are separate questions: a precisely optimised model can still be misspecified or evaluated on the wrong data.

05 / INTERACTIVE EXPERIMENT

Why confident mistakes are expensive

Choose an observed label and change the predicted probability. For y = 1, loss is −log(p); for y = 0, it is −log(1 − p). The graph shows both possibilities, with your selected observation highlighted.

For y = 1, p = 0.9 gives about 0.105 loss, while p = 0.1 gives about 2.303. At p = 0.5, either label gives approximately 0.693. All logarithms here are natural logarithms, so loss is measured in nats.

Numerical stability matters

Computing log(σ(z)) naively can overflow or produce log(0). A stable logit-based expression is ℓ(z,y) = max(z,0) − yz + log(1 + exp(−|z|)). It is mathematically equivalent to softplus(z) − yz but avoids exponentiating a large positive number. Clipping probabilities can protect display code, but production training should use stable implementations.

Regularisation controls coefficient growth

One common objective is J_λ = J + (λ/2)‖w‖², with the intercept unpenalised. Its weight gradient adds λw. L2 discourages large coefficients; L1 can set coefficients exactly to zero. The choice changes the fitted model, not just its presentation. Standardising continuous features makes a shared penalty more comparable across their scales.

In scikit-learn, C controls inverse regularisation strength: smaller C means stronger regularisation. Exact scaling depends on the library’s objective convention, so do not assume C = 1/λ for every handwritten loss. Use the library documentation for solver and penalty compatibility [2].

06 / PROBABILITIES ARE NOT DECISIONS

Move the threshold. Change the consequences.

This example uses 20 illustrative observations with fixed scores to demonstrate how the decision threshold affects predictions. Lowering t can recover more positives, but it can also generate more false alarms. Changing the threshold never changes these probabilities.

Precision = TP/(TP + FP) asks how many positive predictions were correct. Recall = TP/(TP + FN) asks how many actual positives were recovered. Specificity = TN/(TN + FP) describes correctly rejected negatives. When a denominator is zero, this demonstration reports “undefined” rather than pretending a ratio exists.

Recall cannot decrease as the threshold is lowered on fixed data. Precision need not change monotonically. Accuracy can be misleading when one class dominates: if only 1% of cases are positive, always predicting negative achieves 99% accuracy but zero recall.

Choosing a threshold using costs

Suppose a false positive costs C_FP, a false negative costs C_FN, and correct decisions have zero cost. Given an accurate conditional probability p, the expected cost of predicting positive is C_FP(1 − p); for predicting negative it is C_FNp. Comparing the two yields t = C_FP/(C_FP + C_FN). This derivation assumes constant costs and no capacity constraints. Real operating rules may require a different objective. I work through a cost-based business calculation in From Data Quality to AI Value.

Tune decision thresholds on validation data or appropriate cross-validation, then lock the choice before testing. Optimising a threshold on the final test set turns that set into development data. The scikit-learn threshold guide explicitly separates probability estimation from the decision rule [3].

Discrimination and calibration answer different questions

ROC-AUC measures ranking across positive-negative pairs; it does not assess whether a predicted 0.8 corresponds to an 80% event frequency. Precision-recall analysis is often informative when positive retrieval is the operational concern; its baseline and values depend on prevalence. Report the positive class and class distribution rather than treating one metric as universal.

Calibration asks whether groups of similar predicted probabilities have corresponding event frequencies. Reliability plots compare average predictions with observed fractions within bins, but small bins are noisy. Log-loss and Brier score evaluate probability quality, though neither isolates calibration alone. Logistic regression is not automatically calibrated under misspecification, strong regularisation, sampling changes or distribution shift [4].

07 / REPRODUCIBLE PRACTICE

A workflow that keeps the test set untouched

This runnable Python example generates synthetic classification data, reserves validation and test subsets, fits scaling only through the training pipeline, and selects a threshold on validation F1. F1 is a demonstration objective, not a recommendation for every application. A fixed seed supports reproducibility; record Python and package versions alongside any reported results.

import numpy as np
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import (f1_score, roc_auc_score,
    average_precision_score, log_loss, brier_score_loss,
    confusion_matrix)

X, y = make_classification(
    n_samples=2000, n_features=10, n_informative=5,
    n_redundant=2, weights=[0.75, 0.25],
    class_sep=1.0, flip_y=0.03, random_state=42)
X_dev, X_test, y_dev, y_test = train_test_split(
    X, y, test_size=0.2, stratify=y, random_state=42)
X_train, X_val, y_train, y_val = train_test_split(
    X_dev, y_dev, test_size=0.25,
    stratify=y_dev, random_state=42)

model = make_pipeline(
    StandardScaler(),
    LogisticRegression(C=1.0, solver="lbfgs", max_iter=2000))
model.fit(X_train, y_train)
positive_index = list(model.classes_).index(1)
p_val = model.predict_proba(X_val)[:, positive_index]
thresholds = np.linspace(0.05, 0.95, 181)
scores = [f1_score(y_val, p_val >= t, zero_division=0)
          for t in thresholds]
t = thresholds[int(np.argmax(scores))]
# Keep this model and threshold fixed for final evaluation.
p_test = model.predict_proba(X_test)[:, positive_index]
pred = (p_test >= t).astype(int)
print("Chosen threshold:", t)
print("ROC-AUC:", roc_auc_score(y_test, p_test))
print("Average precision:", average_precision_score(y_test, p_test))
print("Log-loss:", log_loss(y_test, p_test))
print("Brier:", brier_score_loss(y_test, p_test))
print("F1:", f1_score(y_test, pred, zero_division=0))
print("Confusion matrix [rows=actual, columns=predicted]:")
print(confusion_matrix(y_test, pred, labels=[0, 1]))

Average precision is a particular summary of the precision-recall curve; it is not always numerically identical to trapezoidal PR-AUC. The example deliberately prints results when run rather than publishing invented scores. It holds C fixed; a fuller study would tune C inside the development data and account for threshold selection in its validation design.

For repeated measurements from the same person, shop or vehicle, use group-aware splits when evaluating generalisation to unseen groups. For future prediction, use a chronological split and construct every feature from information available at prediction time. Random stratified splitting is appropriate only when its exchangeability assumptions match the question.

What to check before trusting the model

• Target timing: define the event, observation window and prediction horizon.
• Leakage: exclude post-outcome features; fit imputation, scaling and feature selection inside training folds.
• Specification: inspect plausible nonlinearities and interactions instead of assuming linear log-odds are adequate.
• Stability: examine convergence warnings, correlated predictors, rare categories and coefficient sensitivity.
• Evaluation: compare with simple baselines; report uncertainty using a resampling scheme consistent with the sampling unit.
• Deployment: monitor prevalence, input distributions, calibration and operational error costs over time.

08 / A RESEARCH PERSPECTIVE

What happens when the data are perfectly separable?

Suppose one hyperplane classifies every training observation correctly. Scaling its coefficients upward pushes positive examples toward probability 1 and negatives toward 0. The unregularised log-loss approaches zero, but generally does not attain zero at any finite coefficient vector. The maximum-likelihood coefficients can therefore diverge even though training classification is already perfect.

Soudry and colleagues studied gradient descent for unregularised logistic regression with homogeneous linear predictors on separable data. Under their stated conditions, the predictor’s direction converges to the hard-margin SVM direction even while its norm grows [5]. This is an example of implicit bias: the optimisation procedure influences which solution is approached. It is not a claim that arbitrary logistic models, optimisers or nonlinear networks always behave this way.

Practically, a low training error is not enough. Track held-out log-loss and coefficient magnitude, use explicit regularisation where appropriate, and distinguish a classification boundary from calibrated probability estimation. This paper provides a theoretical lens, not validation of the demonstrations in this article.

How to turn this tutorial into an empirical study

A useful research question would be: How do regularisation and class imbalance affect discrimination, probability calibration and threshold stability? Predefine the comparison: unregularised or weakly regularised logistic regression, an L2 sweep, and a simple baseline. Generate multiple independent datasets across controlled imbalance and overlap levels, then repeat an identical training/validation/test protocol.

Report distributions of test log-loss, Brier score, ROC-AUC, average precision, selected thresholds and coefficient norms across seeds. Separate changes in the training prevalence from changes in the evaluation population. If class weighting is introduced, treat it as a new objective and check probability calibration rather than assuming its scores retain the original population interpretation.

The key deliverable would be a reproducible experiment with a fixed protocol, uncertainty estimates, failure cases and a discussion of external validity. The sliders on this page support intuition; they cannot substitute for such an experiment.

Three ideas to carry forward

1. The model is linear in log-odds, not in probability.
2. Log-loss trains probability estimates; thresholds govern decisions.
3. A convincing result requires a suitable evaluation design, not just a converged optimiser.