0% found this document useful (0 votes)
9 views5 pages

Logistic Regression Example for Cancer Diagnosis

The document presents a logistic regression analysis for cancer diagnosis using a small dataset with features related to tumor characteristics. It details the process of standardizing features, forming a design matrix, and applying the Newton-Raphson method to estimate model parameters. The analysis highlights issues with small sample sizes leading to unstable estimates and suggests using larger datasets or regularization techniques for more reliable results.

Uploaded by

Wilson Zhuwaki
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
0% found this document useful (0 votes)
9 views5 pages

Logistic Regression Example for Cancer Diagnosis

The document presents a logistic regression analysis for cancer diagnosis using a small dataset with features related to tumor characteristics. It details the process of standardizing features, forming a design matrix, and applying the Newton-Raphson method to estimate model parameters. The analysis highlights issues with small sample sizes leading to unstable estimates and suggests using larger datasets or regularization techniques for more reliable results.

Uploaded by

Wilson Zhuwaki
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd

Worked Numeric Example: Logistic Regression (Cancer

Diagnosis)
Prepared by: Zhuwaki Wilson

Problem and tiny dataset


We model diagnosis as y ∈ {0, 1} where 1 = malignant, 0 = benign. Features are x1 =
radius mean and x2 = area se. Use a tiny example (4 observations):

x1 x2 y
10.0 1.00 0
12.0 1.20 1
9.0 0.80 0
14.0 1.50 1

Standardize features
Compute mean and standard deviation of each column, then standardize:

means = (x̄1 , x̄2 ) = (11.25, 1.125), sds = (s1 , s2 ) = (1.8028, 0.25495).


(std)
Standardized features xj = (xj − x̄j )/sj give:
 
0.4173 0.3431
 0.4159 0.2955 
Xstd = 
−1.2475 −1.2860 (rounded to 4 d.p.)

1.4143 1.6474

Design matrix
Add intercept to obtain the 4 × 3 design matrix:
 
1 0.4173 0.3431
 
1 0.4159 0.2955 
X= 1 −1.2475 −1.2860 .

 
1 1.4143 1.6474

1
Target vector:  
0
1
y=
0 .

Model and notation


Parameters: β = (β0 , β1 , β2 )⊤ . Linear predictor for observation i: z (i) = x̃(i)⊤ β. Probability:
1
p̂(i) = σ(z (i) ) = .
1 + e−z(i)
We solve for β by Newton–Raphson / IRLS.

Iteration 0 (initial guess)


Initialize β (0) = (0, 0, 0)⊤ . Then

z(0) = Xβ (0) = 0, p(0) = σ(z(0) ) = 21 1.

Compute the gradient (score):


 
0
∇ℓ(β (0) ) = X⊤ (y − p(0) ) = 1.8226 (rounded).
 

1.7401

Weight matrix W(0) = diag( p(0) (1 − p(0) ) ) = 14 I. Hence Hessian (negative log-likelihood)
approximation:
 
1.0000 0.0000 0.0000
H(0) = X⊤ W(0) X ≈ 0.0000 1.0000 0.9943 (rounded).
 

0.0000 0.9943 1.0000

(0) (0) (0) (0) (0)


 Solve H ∆ = ∇ℓ(β ) to get the Newton step ∆ . For these numbers ∆ ≈
0
 8.1079 . Update:
−6.3214  
0
β (1) = β (0) + ∆(0) ≈  8.1079  .
−6.3214

2
Iteration 1
Using β (1) compute:
   
−2.2222 0.0978
 1.3333  0.7914
z(1) = Xβ (1) ≈
−1.5556 ,
 p(1) = σ(z(1) ) ≈ 
0.1743 .

2.4444 0.9202

Weight diagonal entries wi = pi (1 − pi ) (rounded):


 
0.0880
0.1651
diag(W(1) ) ≈  0.1438 .

0.0739

Gradient:  
0.0164
∇ℓ(β (1) ) = X⊤ (y − p(1) ) ≈ 0.4637 .
0.4426
Hessian:  
0.4707 −0.0563 −0.0691
H(1) = X⊤ W(1) X ≈ −0.0563 0.4108 0.4109  .
 

−0.0691 0.4109 0.4163


Solve H(1) ∆(1) = ∇ℓ(β (1) ) to get ∆(1) ≈ (0.06519, 5.0569, −3.9180)⊤ . Update:
 
0.06519
β (2) = β (1) + ∆(1) ≈  13.1648  .
−10.2394

Converged estimate (final)


After a few iterations the algorithm converges to
 
1.7417
β̂ ≈  46.0010  (rounded).
 

−33.2850

Predicted probabilities for each observation (final):

p̂ ≈ 5.49 × 10−6 , 0.99996, 3.28 × 10−5 , 0.999999996 .




Thus the model separates the two classes strongly (very small / very large predicted proba-
bilities).

3
Decision rule and decision boundary
Using threshold 0.5, predict malignant when p̂ ≥ 0.5, equivalent to
(std) (std)
β 0 + β 1 x1 + β 2 x2 ≥ 0.

Plugging estimated coefficients gives the decision boundary line in the standardized feature
space:
(std) (std)
(std) β 0 + β 1 x1 1.7417 + 46.0010 x1
x2 = − ≈− .
β2 −33.2850

Odds ratios and inference (note of caution)


Exponentiating coefficients yields odds ratios:

exp(β̂) ≈ 5.71, 9.5 × 1019 , 3.50 × 10−15 .




Approximate standard errors come from the inverse observed information (X⊤ ŴX)−1 .
For this tiny example the estimated standard errors are extremely large (hundreds to thou-
sands) and CIs either explode numerically or become meaningless. This indicates practical
issues:

• The dataset is tiny (only 4 observations).

• The classes are (nearly) separable with these features, which causes large coefficient
magnitudes and unstable standard errors.

• In practice use larger sample sizes, regularization (penalized logistic regression), or


report penalized estimates (e.g., ridge / lasso logistic) to avoid infinite / unstable
MLEs.

Summary of the numeric solve


1. We standardized features and formed the design matrix.

2. Starting at β (0) = 0 we performed Newton–Raphson (IRLS) updates: β ← β −


(X⊤ WX)−1 X⊤ (p − y).

3. Iterations (shown above) rapidly drove coefficients to large magnitudes due to near-
separability.

4. Final β̂ gives near-certain probabilities for the training points; however standard errors
are unreliable.

If you want:

4
• I can add a step-by-step arithmetic expansion of one Newton step showing each
multiply/add in full long form (digit-by-digit),

• or I can replace this tiny dataset with a slightly larger simulated dataset (e.g.,
20–50 observations) and re-run so the coefficients, SEs, and CIs are numerically stable
and interpretable — then produce that full LaTeX output as well.

You might also like