Worked Numeric Example: Logistic Regression (Cancer
Diagnosis)
Prepared by: Zhuwaki Wilson
Problem and tiny dataset
We model diagnosis as y ∈ {0, 1} where 1 = malignant, 0 = benign. Features are x1 =
radius mean and x2 = area se. Use a tiny example (4 observations):
x1 x2 y
10.0 1.00 0
12.0 1.20 1
9.0 0.80 0
14.0 1.50 1
Standardize features
Compute mean and standard deviation of each column, then standardize:
means = (x̄1 , x̄2 ) = (11.25, 1.125), sds = (s1 , s2 ) = (1.8028, 0.25495).
(std)
Standardized features xj = (xj − x̄j )/sj give:
0.4173 0.3431
0.4159 0.2955
Xstd =
−1.2475 −1.2860 (rounded to 4 d.p.)
1.4143 1.6474
Design matrix
Add intercept to obtain the 4 × 3 design matrix:
1 0.4173 0.3431
1 0.4159 0.2955
X= 1 −1.2475 −1.2860 .
1 1.4143 1.6474
1
Target vector:
0
1
y=
0 .
Model and notation
Parameters: β = (β0 , β1 , β2 )⊤ . Linear predictor for observation i: z (i) = x̃(i)⊤ β. Probability:
1
p̂(i) = σ(z (i) ) = .
1 + e−z(i)
We solve for β by Newton–Raphson / IRLS.
Iteration 0 (initial guess)
Initialize β (0) = (0, 0, 0)⊤ . Then
z(0) = Xβ (0) = 0, p(0) = σ(z(0) ) = 21 1.
Compute the gradient (score):
0
∇ℓ(β (0) ) = X⊤ (y − p(0) ) = 1.8226 (rounded).
1.7401
Weight matrix W(0) = diag( p(0) (1 − p(0) ) ) = 14 I. Hence Hessian (negative log-likelihood)
approximation:
1.0000 0.0000 0.0000
H(0) = X⊤ W(0) X ≈ 0.0000 1.0000 0.9943 (rounded).
0.0000 0.9943 1.0000
(0) (0) (0) (0) (0)
Solve H ∆ = ∇ℓ(β ) to get the Newton step ∆ . For these numbers ∆ ≈
0
8.1079 . Update:
−6.3214
0
β (1) = β (0) + ∆(0) ≈ 8.1079 .
−6.3214
2
Iteration 1
Using β (1) compute:
−2.2222 0.0978
1.3333 0.7914
z(1) = Xβ (1) ≈
−1.5556 ,
p(1) = σ(z(1) ) ≈
0.1743 .
2.4444 0.9202
Weight diagonal entries wi = pi (1 − pi ) (rounded):
0.0880
0.1651
diag(W(1) ) ≈ 0.1438 .
0.0739
Gradient:
0.0164
∇ℓ(β (1) ) = X⊤ (y − p(1) ) ≈ 0.4637 .
0.4426
Hessian:
0.4707 −0.0563 −0.0691
H(1) = X⊤ W(1) X ≈ −0.0563 0.4108 0.4109 .
−0.0691 0.4109 0.4163
Solve H(1) ∆(1) = ∇ℓ(β (1) ) to get ∆(1) ≈ (0.06519, 5.0569, −3.9180)⊤ . Update:
0.06519
β (2) = β (1) + ∆(1) ≈ 13.1648 .
−10.2394
Converged estimate (final)
After a few iterations the algorithm converges to
1.7417
β̂ ≈ 46.0010 (rounded).
−33.2850
Predicted probabilities for each observation (final):
p̂ ≈ 5.49 × 10−6 , 0.99996, 3.28 × 10−5 , 0.999999996 .
Thus the model separates the two classes strongly (very small / very large predicted proba-
bilities).
3
Decision rule and decision boundary
Using threshold 0.5, predict malignant when p̂ ≥ 0.5, equivalent to
(std) (std)
β 0 + β 1 x1 + β 2 x2 ≥ 0.
Plugging estimated coefficients gives the decision boundary line in the standardized feature
space:
(std) (std)
(std) β 0 + β 1 x1 1.7417 + 46.0010 x1
x2 = − ≈− .
β2 −33.2850
Odds ratios and inference (note of caution)
Exponentiating coefficients yields odds ratios:
exp(β̂) ≈ 5.71, 9.5 × 1019 , 3.50 × 10−15 .
Approximate standard errors come from the inverse observed information (X⊤ ŴX)−1 .
For this tiny example the estimated standard errors are extremely large (hundreds to thou-
sands) and CIs either explode numerically or become meaningless. This indicates practical
issues:
• The dataset is tiny (only 4 observations).
• The classes are (nearly) separable with these features, which causes large coefficient
magnitudes and unstable standard errors.
• In practice use larger sample sizes, regularization (penalized logistic regression), or
report penalized estimates (e.g., ridge / lasso logistic) to avoid infinite / unstable
MLEs.
Summary of the numeric solve
1. We standardized features and formed the design matrix.
2. Starting at β (0) = 0 we performed Newton–Raphson (IRLS) updates: β ← β −
(X⊤ WX)−1 X⊤ (p − y).
3. Iterations (shown above) rapidly drove coefficients to large magnitudes due to near-
separability.
4. Final β̂ gives near-certain probabilities for the training points; however standard errors
are unreliable.
If you want:
4
• I can add a step-by-step arithmetic expansion of one Newton step showing each
multiply/add in full long form (digit-by-digit),
• or I can replace this tiny dataset with a slightly larger simulated dataset (e.g.,
20–50 observations) and re-run so the coefficients, SEs, and CIs are numerically stable
and interpretable — then produce that full LaTeX output as well.