Advanced Topic

in Regression UNIT 5 BINARY DEPENDENT VARIABLE

Analysis
MODELS*

Structure

5.0 Objectives
5.1 Introduction
5.2 Nature of Variables
5.3 The Linear Probability Model (LPM)
5.3.1 Non-Normality of Disturbances
5.3.2 Heteroscedastic Variances
5.3.3 Non-Fulfilment of Probability Limits
5.3.4 R2 as Measure of Goodness of Fit
5.3.5 Limitations of LPM and Corrective Measures

5.4 The Logit Model

5.4.1 Properties of Logit Model
5.4.2 Estimation of Logit Model
5.4.3 Interpretation of Logistic Parameters
5.4.4 Application of Logit Model for Grouped Data

5.5 The Probit Model

5.5.1 Empirical Example of Probit Model
5.5.2 Estimation of Marginal Effect in Probit Model

5.6 Let Us Sum Up

5.7 Key Words
5.8 Suggested Books for Further Reading
5.9 Answers/Hints to Check Your Progress Exercises

5.0 OBJECTIVES
After reading this unit, you will be able to:

• specify the nature of variables in the ‘binary dependent variable models’;

• define a ‘linear probability model (LPM)’ with an illustration;
• discuss why the OLS method of estimation is not applicable in case of ‘binary
dependent variable models’;
• state why the conventional R2 is not of use in case of ‘binary response models’;
• indicate the form of a Logit model indicating why the OLS method of
estimation cannot be applied here;

80 *
Prof. Sushil Haldar, Jadavpur University.
• list the properties of a Logit model; Binary Dependent
Variable Models
• outline the estimation procedure for the parameters of a Logit model;
• explain the interpretation of Logistic parameters;
• illustrate the interpretation of logistic parameters with an empirical exercise;
• write a short note on the application of Logit model for ‘grouped data’;
• describe the application of Probit model with an illustration for the estimation
of its ‘marginal effect’;
• compute the ‘marginal effect’ in an estimated Probit model for a ‘mean value’;
and
• present the relationship between the regression coefficients of: (i) Logit model
and Probit model and (ii) LPM and Logit model.

5.1 INTRODUCTION
In applied economics, we face situations where the ‘dependent variable’ is
binary or dichotomous or qualitative. Some examples are as follows: Banks
provide loans to the customers. Some customers may be defaulters who fail
to re-pay their loans to the banks in time. Therefore, the borrowers can be
divided into two groups: defaulters and non-defaulters. In medical science, a
group of people of a fixed age cohort is diagnosed about the incidence of
diabetes. Some persons may suffer from diabetes while others may not. Thus,
two groups of persons emerge – diabetic and non-diabetic. Numerous other
examples can be given like: vegetarian and non-vegetarian, smokers and non-
smokers, drug addicted vs. non-drug addicted, owning a car or not, etc. In all
these cases, the dependent variable is qualitative and can be quantified as 1
and 0. Now the question that arises is about the factors that influence the
probability of being, say, a defaulter. Is it due to the age of the customer? Or
is it due to the cultural characteristics? Or is it due to the marital status? Or is
it due to the overall socio-economic profile? We can similarly raise many
questions pertaining to the incidence of diabetes, smokers vs. non-smokers,
vegetarian vs. non-vegetarian, etc. In most cases of regression analysis, we
consider the dependent variable as quantitative or numeric whereas the
independent variables may be either quantitative or qualitative or a mixture of
both the types. In this Unit, we consider the dependent variable as not only
qualitative but also as dichotomous. For instance, if the dependent variable is
‘yes’ or ‘no’ type, then we define the dependent variable as binary (viz. 1 or
0) and proceed.

5.2 NATURE OF VARIABLES

In two variable regression models, we try to establish an average
mathematical relationship between the two variables. The dependent variable
is termed in various ways (e.g. regressand or endogenous or response or
explained) whereas, the independent variables are termed as regressors (or
81
Advanced Topic explanatory or exogenous variables). Let us now suppose that we want to
in Regression
Analysis study the ownership of a car of an individual. Car being a luxury good,
owning it depends on the income level. It may also depend on where the
person resides i.e. in urban or rural areas. In spite of such factors, a person
may still own a car or not. We therefore define as 1, if the person owns a car,
0 otherwise. The two variables regression equation can then be written as:
yi =α + β xi + ui ........................... (5.1)

where, yi = 1 if owning a car; yi = 0 otherwise. Here, α and β are the

parameters to be estimated given the level of income xi . Now, is the standard
‘method of least squares’ applicable? What are the possible consequences if
we apply the ‘method of least squares’? These issues are dealt with in this
section. In a model where yi is quantitative, our objective is to estimate its
expected or mean value given the values of the explanatory variables. The
regressor(s) can either be quantitative or qualitative. If the regressor is
qualitative [like locality of dwelling (rural or urban), gender (male or female),
religion (Hindu or Non-Hindu), caste (backward or not-backward), etc.] we
can estimate the influence of such categorical variables on the dependent
variable by considering them as dummy variables. This means, both the
dependent as well as independent variables may be simultaneously,
dichotomous. In such cases, Equation (5.1) can be written as:

y i = α 0 + α 1 Di + u i (5.2)

where, Di = 1 if the person resides in urban area, 0 otherwise. The dependent

variable (viz. owning a car) remains the same. The parameter α1 determines
the influence of residential status on having a car or not. We can incorporate
more variables on the right hand side. For instance, we can include both the
income and residential status as follows:
y i = α + β x i + α 1 Di + u i (5.3).
In models where y is qualitative, our objective is to find the probability of
something happening (e.g. owning a car or owning of a house or belonging to
any category or group). Such qualitative response regression models are
known as ‘probability models’. In this unit, we discuss three alternative
approaches for estimating the parameters in such binary response models.
These are: (i) The Linear Probability Model (LPM), (ii) The Logit Model and
(iii) The Probit Model.

5.3 THE LINEAR PROBABILITY MODEL (LPM)

In order to explain the linear probability model (LPM), we consider a model
like:
y i = β 0 + β 1 xi + u i (5.4)
82
where, xi is the income (independent variable) and yi is the dependent Binary Dependent
Variable Models
variable referring to the ownership of a car. Note that Equation (5.4) has the
conditional expectation of yi given xi i.e. �(�� |�� ). This is interpreted as the
conditional probability that the event yi will occur given xi , that is
‘probability of (�� |�� ). This is the reason why a model like (5.4) is also
known as the ‘Linear Probability Model (LPM)’. Hence, the expectation of
owning a car given a certain level of income can also be written as:

E ( yi / xi ) =β 0 + β1 xi + ui (5.5a)

Now, if pi is the probability that yi =1 and 1 − pi is the probability that yi = 0 ,

the variable yi has its probability distribution as in Table 5.1.

Table 5.1: Probability Distribution of yi

yi Probability
0 1 – pi
1 pi
Total 1

Thus, yi follows the Bernoulli Distribution. By applying mathematical

expectation, we get:

E ( yi ) = 0.(1 − pi ) + 1. pi = pi or

E ( y i | xi ) = β 0 + β 1 xi = p i (5.5b)

Extending the above to ‘n’ independent trials, each with probability ‘p’ for
success and probability (1–p) for failure, and if X of these trials represent the
number of successes, then we know that X follows the binomial distribution
with mean ‘np’ and variance ‘npq’ where q=(1–p). Since the probability p i
must lie between 0 and 1, we have the restriction:

0 ≤ E ( y i | xi ) ≤ 1 (5.6)

Equation (5.6) means that the conditional expectation (or conditional

probability) of yi lies between 0 and 1. Let us now examine whether the
ordinary least squares (OLS) method for estimating the model (5.5) works.
We will see that we encounter three specific types of problems viz. (i) non-
normality of the disturbances u i , (ii) heteroscedastic variances of the
disturbances and (iii) non-fulfilment of conditional probability: 0 ≤ �(�� |�� .

5.3.1 Non-Normality of Disturbances

Recall that OLS does not require the disturbances ( u i ) to be normally
distributed. However, we need to examine ‘normality’ for the purpose of

83
Advanced Topic ‘statistical inference’. Note that like yi , the disturbances u i also can take only
in Regression
Analysis two values which follow Bernoulli distribution as shown in Table 5.2.

Table 5.2: Probability Distribution of ui

yi ui Probability
When yi =1 1 − β 0 − β 1 xi pi
When yi = 0 − β 0 − β 1 xi 1 − pi

Total - 1

Hence, u i also follows Bernoulli distribution. Therefore, as the sample size

increases indefinitely, the OLS estimators of Equation (5.5) tend to be
normally distributed. As a result, in large samples, the statistical inference of
the LPM can be made by estimating the parameters of (5.5) by the OLS
procedure (under the normality assumption). However, the non-normality
continues to prevail as can be seen from Fig.5.1 in which vertically we
measure y (which takes the two values 1 and 0) and horizontally we measure
x. The expected value of y, or the conditional probability, is shown by the
line E[y‫׀‬x]. We can see from the Figure that the error term is not normally
distributed since it has two possible values for given value of x i.e. one value
given by A (when y = 1) and the other value given by B when y = 0. At point
A, y = 1 and ui = ‘observed value – predicted value’ or:
u i = 1 − Pr[ y | x] = 1 − β 0 − β1 xi . Similarly, at point B, the value of y being
zero, it yields, u i = 0 − β 0 − β 1 xi . The curve has no scope of becoming bell
shaped. Hence, ui is not normally distributed.

Fig. 5.1: Graphical Presentation of a LPM

5.3.2 Heteroscedastic Variances

Even if E (ui ) = 0 and cov (u i , u j ) = 0 for i ≠ j (i.e. no serial correlation), it
cannot be maintained that in the LPM the disturbances are homoscedastic. As
statistical theory shows, for a Bernoulli distribution the theoretical mean and
variance are p and p (1 − p) respectively ( p being the probability of success).
This shows that the variance is a function of the mean. Hence, the variance of
error terms are heteroscedastic. This can also be formally seen. Since the
84 probability distribution of ‘ui’ is as in Table 5.2, we have:
 Var (ui ) = (1 − β 0 − β1 xi ) 2 . pi + (− β 0 − β1 xi ) 2 .(1 − pi ) Binary Dependent
Variable Models
=1 − 2( β 0 + β1 xi ) + ( β 0 + β1 xi ) 2  . pi + ( β o + β1 xi )2 − ( β 0 + β1 xi ) 2 . pi
=pi − 2( β 0 + β1 xi ). pi + ( β 0 + β1 xi ) 2 . pi + ( β o + β1 xi ) 2 − ( β 0 + β1 xi ) 2 . pi
=pi − 2( β 0 + β1 xi ). pi + ( β o + β1 xi ) 2
= pi .(1 − pi )

Since from (5.5b), �� + �� �� = �� (5.7)

It follows from Equation (5.7) that the expected value of ui is a function of x

and hence not a constant. This is because ‘the expected value of ui is
contained in the Var (ui). This tells us that the Var (ui) also depends on the
values of x and hence is not homoscedastic. Because of this
heteroscedasticity, the OLS estimators will be unbiased but inefficient i.e.
they do not have the minimum variance property. One way to resolve the
problem of heteroscedasticity is to transform the Equation (5.4) by dividing it
throughout by pi (1 − pi ) = wi .

By doing this, we get:

yi β βx u
= 0 + 1 i + i ................................(5.8)
wi wi wi wi

The error term in (5.8) will now be homoscedastic. Therefore, after

estimating Equation (5.4), we can estimate Equation (5.8) by OLS. This is
known as the ‘weighted least squares (WLS)’ method where wi serves as the
weights. But in practice, the true E ( y i / xi ) is unknown. Hence, the weights
wi are also unknown. To estimate wi , we can proceed as follows. Run the
OLS regression in Equation (5.4) despite the presence of heteroscedasticity
∧ ∧ ∧ ∧
and obtain y i = E ( yi | xi ) . Then obtain =
wi y i (1 − y i ) as the estimate of wi .
Use this wi to transform the initial equation and obtain the new transformed
equation. We can now estimate the transformed equation by OLS (WLS).

5.3.3 Non-fulfilment of Probability Limits

Since E ( y i / xi ) in the linear probability models measures the conditional
probability of the event yi occurring given xi , it must necessarily lie
∧
between 0 and 1. Although this is true a priori, there is no guarantee that y i ,
the estimators of E ( y i / xi ) , will necessarily fulfill this condition. This is the
real problem with the OLS estimation of the LPM. There are two ways of
∧
finding out whether the estimated y i lie between 0 and 1. One is to estimate
∧
the LPM by the usual OLS method and find out whether the estimated y i lie
∧
between 0 and 1. If some values are less than 0 (that is, negative), y i is
assumed to be zero for those cases; if they are greater than 1, they are
85
Advanced Topic assumed to be 1. The second procedure is to devise an estimating technique
in Regression ∧
Analysis that will guarantee that the estimated conditional probabilities y i lie between
0 and 1. The Logit and Probit models (discussed later) will guarantee that the
estimated probabilities will indeed lie between the logical limits of 0 and 1.

5.3.4 R2 as Measure of Goodness of Fit

The conventional R 2 is of limited value in the Binary or dichotomous
response models where corresponding to a given X, Y is either 0 or 1.
Therefore, all the Y values will either lie along the X axis or along the line
corresponding to 1 (as shown in Fig. 5.1). Therefore, the LPM is not
expected to fit such a scatter well. But if a LPM is estimated in such a way
that it will not fall outside the logical band 0–1, the conventional R 2 is likely
to be much lower than 1 for such models. Therefore, in practical applications,
R 2 is of little use in case of binary dependent models.

5.3.5 Limitations of the LPM and Corrective Measures

The LPM suffers from non-normality of u i , heteroscedasticity of u i ,
possibility of the predicted value of ‘y’ lying outside the 0–1 range and a very
low value of R2. However, we can use WLS to resolve the heteroscedasticity
problem and increase the sample size to minimise the non-normality problem.
By resorting to restricted least-squares or mathematical programming
techniques, we can even make the estimated probabilities lie in the 0–1
interval. But, a fundamental problem with the LPM is that it is not logically a
very attractive model because it assumes that pi = E ( yi | xi ) increases
linearly with xi i.e. the marginal or incremental effect of x remains constant
throughout. This is unrealistic. In reality, one would expect that p i is
nonlinearly related to xi. This is because, at very low income a family will not
own a car. But at a sufficiently high level of income, say x * , it is most likely
to own a car. Any increase in income beyond x * will have little effect on the
probability of owning a car. Thus, at both ends of the income distribution, the
probability of owning a car will be virtually unaffected by a small increase in
income (x). Therefore, what we need is a (probability) model that has these
two features: (i) as xi increases, pi = E ( yi | xi ) should also increase but not
step outside the 0–1 interval, and (ii) the relationship between p i and xi is
nonlinear i.e. “one which approaches ‘zero’ at slower and slower rates as xi
gets small” and “one which approaches ‘one’ also at slower and slower rate
as xi gets very large”. Such a model is the Logit model discussed next.

Check Your Progress 1 [answer within the space given in about 50-100
words]

1) What is the nature of variables we will have in a ‘binary dependent

variable model’?
……………………………………………………………………………
86
 …………………………………………………………………………… Binary Dependent
Variable Models
……………………………………………………………………………
……………………………………………………………………………
……………………………………………………………………………
2) Can the method of OLS be applied for estimating the parameters of a
‘binary dependent variable model’? Give reasons.
……………………………………………………………………………
……………………………………………………………………………
……………………………………………………………………………
……………………………………………………………………………
……………………………………………………………………………
3) What method is applicable to overcome the problem of
heteroscedasticity in estimating a binary dependent variable model like
in Equation (5.4)?
……………………………………………………………………………
……………………………………………………………………………
……………………………………………………………………………
……………………………………………………………………………
……………………………………………………………………………
4) Why is R2 not a good measure of fitness of the model in case of binary
dependent variable models?
……………………………………………………………………………
……………………………………………………………………………
……………………………………………………………………………
……………………………………………………………………………

5.4 THE LOGIT MODEL

The logistic regression, or the Logit model, overcomes the limitations of
LPM in case of dummy dependent variable models. We continue with the
conditional expectation of y as in the LPM. That is:
pi = β 0 + β 1 xi (5.9)

where xi is income and pi = E ( yi | xi ) is the conditional probability of the

individual owning a car. Now, consider the following representation of car
ownership between income and probability of owning a car:

1 1 ez
pi = = = (5.10)
1 + e −( β 0 + β1 xi ) 1 + e −z 1 + e z 87
Advanced Topic where, z = β 0 + β 1 xi so that if z tends to –∞, p tends to zero and if z tends to
in Regression
Analysis ∞, p tends to 1. Hence, zi ranges from − ∞ to + ∞ and p i ranges between 0
and 1. Here, p i is nonlinearly related to z i . This is an important property of a
logistic Equation like (5.10). Graphically, this is as shown in Fig. 5.2.

Fig. 5.2: Logistic Curve

The question now is, can we apply the OLS to estimate the parameters of the
Equation (5.10)? The answer is ‘no’. This is because p i is nonlinear not only
in x but also in the β’s. This means we cannot use the OLS procedure to
estimate its parameters. If p i is the probability of owning a car, then (1 − pi )
is the probability of not owning a car. Therefore, we consider the ratio of
pi
odds (or the ‘odds ratio’) as and consider estimating ‘1 – pi’. This
1 − pi
means, we estimate:
e zi
1 − pi =1 −
1 + e zi
1 + e z i − e zi
1 − pi =
1 + e zi
1
1 − pi = zi .........................................................(5.11)
1+ e
Equation (5.11) gives us the probability of not owning a car. Therefore, we
can write the ‘odds ratio’ as:

e zi
pi 1=+ e zi e zi .......................................................(5.12)
=
1 − pi 1
1 + e zi

The ratio of the probabilities as in Equation (5.12) is known as ‘odds ratio in

favour of owning a car’. This is the ‘ratio of the probability that an individual
will own a car to the probability that he will not own a car’. Numerically, this
means, if pi = 0.8 , the odds are ‘4 to 1’ [since pi/(1 – pi) = 0.8/0.2 = 4] in
favour of the individual owning a car. Now, taking the natural log of (5.12),
we get:
pi
Li ln(
= = e zi zi
) ln=
1 − pi
88 L=
i z=
i β 0 + β1 xi ....................................................................(5.13)
In (5.13), Li is the log of odds ratio. It is linear in xi . It is also linear in the Binary Dependent
Variable Models
parameters. Li or (5.13) is called the ‘Logit model’.

5.4.1 Properties of Logit Model

Can we directly estimate Equation (5.3) or Li if pi = 0 and 1? The answer is
‘no’. This is because, if p i = 0, Logit ( Li ) is undefined. Likewise, if we p i
=1, Li is infinity. This poses the question, what is the limiting value of Li?
Since, Zi and Li are both same, and Z lies between –∞ < Z < +∞ (from the
logistic curve), Li also lie between –∞ and +∞. Therefore, as p i goes from 0
to 1, the Logit Li goes from − ∞ to + ∞ . That is, although the probabilities lie
between 0 and 1, the Logit is differently bounded. This can be explained as
follows.
pi
We have,=Li ln( = ) ln= e zi zi .
1 − pi
p
Hence: i = e zi ...............................................(5.14)
1 − pi

where, if z i → − ∝, pi → 0 and if z i →∝, pi → 1 . Thus, although Li is linear

in xi , the probabilities themselves are not. If Li i.e. the Logit is positive, it
means when the value of regressor increases, the odds that the regressand
equals 1 (i.e. y = 1 or some event of interest happens) increases. In order to
explain this further, let us assume, that the value of Li = 0.7. This means
p p
ln = 0.7 ⇒ = e 0.7 = (2.71) 0.7 = 2.009 . This is greater than 1.
1− p 1− p
Hence, the odds in favour of ‘happening of the event’ increases as the
regressor (x) increases. This can be interpreted in a different way. One can
verify that the value of p = 0.667 as follows:
p p
ln = 0.7 ⇒ = e 0.7 = (2.71) 0.7 = 2.009
1− p 1− p

p = 2.009 − 2.009 p
⇒ p(1 + 2.009 ) = 2.009
2.009
⇒ p=
3.009
⇒ p = 0.667

This means that the probability of the event’s (y = 1) happening, is 66.7

percent, which is higher than 50 percent. Now, let us see how we can
interpret the Logit, when it is negative, say Li= -0.7? This means:
p p
ln = −0.7 ⇒ =e−0.7 =(2.71) −0.7 =
0.497. This is less than one. This
1− p 1− p
means the odds in favour of the event’s (y = 1) happening decreases as the
regressor (x) increases. This can also be interpreted in probability terms. The
89
Advanced Topic probability of the event’s is happening 33.19 percent (as can be seen by the
in Regression
Analysis following calculations) if the regressor increases. Given that:
p
= 0.497
1− p
⇒ p = 0.497 − 0.497 p
⇒ p(1 + 0.497) = 0.497
0.497
⇒ p=
1.497
⇒ p = 0.331

This is also less than 0.5 or 50 percent! Hence, if L is negative, the odds that
the regressand equals 1 decreases as the value of x increases. To put it
differently, the Logit becomes negative and increasingly large in magnitude
as the odds ratio decreases from 1 to 0. It also becomes increasingly large and
positive as the odds ratio increases from 1 to infinity. More formally, the
interpretation of the Logit model given in (5.13) is as follows: β 1 , the slope,
measures the change in Li for a unit change in xi i.e. it tells us how the log-
odds changes in favour of owning a car as income changes by a unit. The
intercept β 0 is the value of the log odds in favour of owning a car if income
is zero. Given a certain level of income, say, x * , we can estimate the odds in
favour of owning a car if β 0 and β 1 are known. Here, we have considered
only one regressor but one can add as many regressors (including categorical)
as may be dictated by the underlying theory.

5.4.2 Estimation of the Logit Model

Let us add the error term in Equation (5.13). We get:
pi (5.15)
Li ln( ) 0 x
1 i ui
1 pi

In order to estimate the parameters, we need data. Data may be of two types
i.e. either for a group or at the individual level. Here, we mainly consider the
individual level data. If we have data at individual level, OLS estimation of
the above equation is infeasible. This can be explained as follows. Consider
our earlier case of owning a car. In terms of the data at the individual level,
p i = 1 if the individual owns a car and p i = 0 if he does not own. But if we
1
put these values directly into the Logit ( Li ) we obtain, Li ln which is
0
1
infinite if the individual owns a car and Li ln which is undefined for the
0
case of individual not owning a car. These expressions are therefore
meaningless. Hence, if we have data at the micro, or individual level, we
cannot estimate (5.15) by the standard OLS method. In this situation, we have
to resort to the ‘maximum-likelihood (ML)’ method of estimating the
parameters. In our earlier logistic Equation (5.10), we assumed that the
90
probability of owning a car follows logistic equation. We cannot observe pi, Binary Dependent
Variable Models
rather we can only observe y = 1 if the individual owns a car, 0 otherwise.
Since, yi is a Bernoulli random variable, we can write Pr( yi = 1 | x) = pi and
Pr( yi = 0 | x) = 1 − pi . Now, suppose we have a random sample of ‘n’ number
of observations. Letting f i ( y i ) denote the probability that yi = 1 or 0; the joint
probability of observing the ‘n.y’ values i.e. f ( y1 , y 2 ....... y n ) is given by:
n n
f ( y1 , y 2 ....... y n ) = ∏ f i ( y i ) = ∏ piyi .(1 − p i )1− yi (5.16)
1 1

We write the ‘joint probability density function’ as a product of individual

density functions because each yi is drawn independently and each yi has the
same logistic density function. The joint probability as given in equation
(5.16) is known as the ‘likelihood function’. In (5.16), we have skipped the
constant term of the Bernoulli distribution (like NCX in binomial distribution)
because it is independent of response or outcome variable (like yi in our case).
Now, we take log of (5.16) and denote it as ‘Log Likelihood Function (LLF)’
to get:
n
LLF yi .ln pi (1 yi ).ln(1 pi )
1
n
yi .ln pi yi .ln(1 pi ) ln(1 pi )
1

n
pi n
yi .ln ln(1 pi )
1 1 pi 1

n n
1 xi )
yi 0 1 xi ln 1 e( 0
(5.17)
1 1

 p 
Thus, the values of ln i  and ln(1 − pi ) are estimated as ( β 0 + β 1 xi ) and
 1 − pi 
− ln(1 + e β 0 + β1 xi ) respectively. The objective is to maximise the LLF i.e.
Equation (5.17) with respect to β0 and β1 using the values of x which are
known. The resulting solutions become non-linear in the parameters because
of the presence of e β 0 + β1 x term. This is the reason why the OLS method
cannot be applied for the Logit model. Once the values of the parameters are
known, we can easily estimate the logistic equation.

5.4.3 Interpretation of Logistic Parameters

Once the Logit model is estimated, the most important part is to interpret the
estimated parameters. For this, we will proceed to analyse the Logit
coefficients, by continuing with the earlier example of ownership of car.
Though one can include more explanatory variables in the Logit model, we
restrict ourselves to considering only one independent variable (i.e. level of
income). We further assume that the estimated Logit model is:

Lˆ = βˆ0 + βˆ1 xi (5.18)

91
Advanced Topic where, β̂ 0 and βˆ1 are the estimated parameters. If income (say, xi) is zero,
in Regression
Analysis then L̂ is β̂ 0 . This means the log of odds is β̂ 0 if income is zero. How do we
interpret this? Now, assume that the value of β̂ 0 is 1. Then, the odds in favour
of owing a car is: 2.71! This is higher than 1. What happens to log of odds or
odds if the estimated value of β̂ 0 is –1? The odds in favour of owning a car is
reduced to 0.369 if β̂ 0 becomes –1! This can be verified as follows: From
(5.14), if income xi is zero, then Lˆ , the estimated L can be written as:
.
 p   p 
Lˆ = ln
ˆ
 = β̂ 0 . Applying antilog, this becomes   = e β 0 . We know
1− p  1− p 
the value of e is approximately 2.71. Given that βˆ = 1 , this means the odds
0

in favour of happening the event is 2.71, which is greater than 1. In the same
way, if we consider the value of βˆ 0 = −1 , the odds is:

 p 
 = e β 0 = (2.71)−1 = 0.369
ˆ

1− p 

which is less than 1. Again, how do we interpret the odds if β̂ 0 is zero? The
odds of owning a car becomes 1 if the value of β̂ 0 = 0! We need to explain
this. From (5.14), if income xi is zero, then Lˆ , the estimated L is written as:
.
 p   p 
Lˆ = ln
ˆ
 = β̂ 0 . This can be written as:   = e β 0 . The left hand side
1− p  1− p 
represents the odds ratio in favour of owning a car. The right hand expression
is known since the value of e is 2.71 and β̂ 0 is known. If β̂ 0 = 0, the odds
ratio in favour of owning a car is 1. This means p = (1 − p ) i.e. the probability
of owning a car is equal to the probability of not owning a car! This means, at
a certain level of income, one may be indecisive or indifferent. Now, let us
proceed to interpret the estimated coefficient of other parameter βˆ1 . If we
ˆ
differentiate (5.16) with respect to xi, we obtain: ∂L = βˆ1 . This gives us the
∂xi
incremental increase in the log of odds in favour of owning a car due to
increase in one additional unit of income (xi). Like in the previous case, if we
know the exact estimated value of βˆ1 , we can calculate the odds ‘in favour of
owning a car’. Let us take the estimated value of βˆ = 0.8. The odds in favour
1
0.8
of owning a car will be 2.22 (=e ), which is greater than 1. What happens to
the odds in favour of owning a car if βˆ1 becomes –0.8? The odds in favour of
owning a car will be 0.45 which is less than 1. Can we estimate the marginal
effect given the exact estimated value of βˆ1 ? Marginal effect means
incremental increase in p due to one unit increase in explanatory variable
∂pi
( x ). Mathematically, we can write: Marginal Effect = . Differentiating
∂xi
92
Equation (5.16) with respect to xi and using the fact that Binary Dependent
Variable Models
  p 
 βˆ0 + βˆ1 xi =
ln    which follows from the Equation (5.13), we get:
  1 − pi  

Lˆ ˆ ˆ x p ˆ
0 1 i ln 1
xi xi 1 pi
1 pi ˆ
.
pi xi 1 pi
1

1 pi
pi pi
1 pi . pi. .
1 pi xi xi ˆ
2 1
pi 1 pi

pi
xi ˆ
1
pi 1 pi
pi ˆ . p (1
1 i pi) ( 5 .1 9 )
xi

The above expression stands for ‘marginal effect’. It basically captures the
incremental change in the probability of owning a car ( y = 1 ) due to one unit
increase in xi . But the major problem arises about the estimation of pi .
Theoretically, p i should be n values (i = 1, 2, 3…………n) but it is
unrealistic to estimate n number of marginal effects! There should be only
one p . Note that ‘regression’ basically captures average mathematical
relationship between two variables as is our case here. Therefore, we have to
estimate that value of p which corresponds to the ‘mean predicted
probability’. The unique value of p can be solved from the estimated Logit
model as follows. From 5.13, we can write: Lˆ = βˆ0 + βˆ1 xi . Replacing xi by its

mean value, x =
∑ x , the new estimated Logit becomes: Lˆ = βˆ + βˆ x
0 1 .
n
Now, the right hand side is known. Let us set βˆ + βˆ x = C. We then have: 0 1

 pˆ 
Lˆ =C ⇒ ln  =C
 1 − pˆ 
pˆ
⇒ eC =
= k
1 − pˆ
pˆ
⇒ =k
1 − pˆ
k
⇒ pˆ = ( pˆ = k (1 − pˆ ); pˆ = k − kpˆ ; (1 + k ) pˆ = k ; pˆ = k / 1 + k )
1+ k
⇒ pˆ =p*

93
Advanced Topic k
in Regression Thus, assuming, eC = k , we obtained p* = . This is the value of ‘mean
Analysis
1+ k
predicted probability’. By doing this, instead of multiple values of p, we have
estimated a single p = p * . Thus, from Equation (5.19), the ‘marginal effect’
can be obtained by the following equation:
∂p ˆ *
Marginal Effect = = β 1 . p (1 − p * ) (5.20)
∂x

Therefore, in order to estimate the ‘marginal effect’ we need two

information: (i) estimated Logit model and (ii) the mean value of the
regressor. A numerical example will be helpful to grasp the problem.
Consider the example of ownership of car of some 25 individuals whose
estimated Logit model is given by: Lˆ = βˆ0 + βˆ1 xi =
−0.75 + [Link] . For
simplicity, we further assume that the mean of x = x = 100 . Then the Logit at
the mean value is: Lˆ = −0.75 + 0.01×100 = 0.25. We know that
 p 
*
Lˆ = ln  , where p * is the ‘mean predicted probability’ evaluated at
* 
1− p 
 p*  0.25
mean value of x. Thus,  = * 
2.71
= 1.28. This yields p* = 0.56
 1− p 
(since 1.28/(1 + 1.28) = 1.28/2.28 = 0.56). Now, using Equation (5.20), we
can estimate the ‘marginal effect’ as:
∂p ˆ *
Marginal Effect = = β1 . p (1 − p * ) = 0.01 × 0.56 × 0.44 = 0.024 . This
∂x
means, the probability of owning a car on an average will increase by 2.4
percent due to one unit increase in the level of income. To illustrate further,
let us now consider a practical example as in Table 5.3.

In the Table, if an individual owns a car, then it is denoted as y = 1, if he does

not own, y = 0. The income is measured in Rupees (thousand per month). We
run the Logit model and obtain the following result as in Table 5.4.

Table 5.3: Illustrative Data for Y and X

(X is in Rs. ‘000)

Individual Y X Individual Y X

1 0 30 14 1 70

2 1 55 15 1 65

3 1 48 16 0 38

4 0 22 17 0 40

5 0 24 18 1 45

94 6 1 50 19 0 38
 Binary Dependent
7 1 60 20 1 70 Variable Models

8 1 55 21 0 56

9 0 40 22 1 48

10 0 38 23 0 40

11 1 60 24 1 75

12 0 30 25 0 38

13 0 45

Table 5.4: Results of Logistic Regression

Logistic regression Number of obs = 25

LR chi2(1) = 21.56 Prob > chi2 = 0.0000
Log likelihood = -6.5262 Pseudo R2 = 0.622
----------------------------------------------------------------------------------------
Dep. Var. Logit Coef. Std. Err. Z P>|z|
----------------------------------------------------------------------------------------
Income(x) 0.293 0.1218 2.41 0.016
Constant –13.76 5.6374 –2.44 0.015

The estimated Logit model is:

Lˆ = −13.76 + 0.293xi (5.21)

We can interpret the results of Equation (5.21) as follows. If income is zero,

the value of log of odds ratio is ‘–13.76’. This means the odds of owing a car
is almost zero (since it is 1.05 × 10−06 ) . The coefficient of xi (viz. 0.293) is the
log of odds of owning a car. It is positive. But we must estimate odds in
favour of owning a car. The odds ratio in favour of owning a car is 1.33. This
can be verified as follows:

The estimated Logit model is: Lˆ = −13.76 + 0.293xi .

 p 
Here, Lˆ = ln  = −13.76 + 0.293xi . Differentiating this with respect to
1− p 
xi, we get:

∂Lˆ
= 0.293
∂xi
p p
= (2.71)
0.293
⇒ = e 0.293 ⇒
1− p 1− p
p
⇒ = 1.33
1− p

95
Advanced Topic This is higher than 1. Therefore, the odds in favour of owning a car increases
in Regression
Analysis as income increases. To calculate the marginal effect, we first estimate the
mean of x which is x = 47.2. By using this in Equation (5.21), we get:

Lˆ = −13.76 + 0.293 × 47.2 = 0.069

p
⇒ = e 0.069
1− p
p
⇒ = 2.710.069
1− p
p
⇒ = 1.071
1− p
⇒ p = 0.517

Therefore, the mean predicted probability is p = p* = 0.517 . Now, using this

estimate of ‘p’ in Equation (5.20), we can estimate the ‘marginal effect’ as:
∂p ˆ *
= β1 . p (1 − p * ) = 0.293 × 0.517 × 0.483 = 0.073 . This means that one unit
∂x
increase in income (x) increases the probability of owning a car by 7.3
percent. The question now is: are the two coefficients statistically significant?
The answer is ‘yes’. The z values are high and both the parameters are found
to be significant at 1 percent level (see the last two columns of Table 5.4).
We have already discussed about R2, as a measure of goodness of fit’ in case
of LPM. The R2 is assumed to be least important in case of categorical
dependent variable. There are various concepts of R2 which are developed
like: Pseudo R2, McFadden R2 and count R2 . The count R2 is the simplest
CP
form of measuring goodness of fit. It is defined as: Count − R 2 = , where
N
CP = number of correct predictions and N = total number of observations.

5.4.4 Application of Logit Model for Grouped Data

Let us now consider the data on several individuals who form groups
according to their income level and the number of individuals owning a car at
each income level. Corresponding to each income level xi , there are N i
individuals with ni among them owning a car. Therefore, we can compute the
probability of owning a car as:

ni
pi (5.22)
Ni

This is the relative frequency. Hence, we can use it as an estimate of the true
p i corresponding to each xi . If N i is fairly large, p̂i will be a reasonably
good estimate of p i . Using the estimated p i , we can obtain the estimated
Logit as:

pi
Li ln( ) 0 1 xi (5.23)
96 1 pi
Equation (5.23) is expected to be a fairly good estimate of the ‘true Logit’ if Binary Dependent
the number of observations Ni at each xi is reasonably large. Variable Models

5.5 THE PROBIT MODEL

The Logit model uses the cumulative logistic function as shown in Equation
(5.13). But this is not the only ‘cumulative distribution/density function’
(CDF) that one can use. In some applications, the normal CDF is used for
‘limited dependent variable models’. We know that the probability density
function (PDF) and the cumulative density function (CDF) of normal
distribution are respectively given by:
2 2
 X −µ   X −µ 
1 − 0.5   X 1 − 0. 5  
PDF = f ( x) = .e  σ 
CDF = F ( x ) = ∫ .e  σ 

σ 2π −∝
σ 2π
The econometric model based on the ‘dummy dependent variable’ that
emerges from the ‘normal CDF’ is popularly known as the Probit model. We
can explain the Probit model using the car ownership example of individuals.
Owning a car gives some satisfaction, which is an unobserved or latent
variable. We assume that the decision of the ith individual to own a car
depends on an unobserved utility level, Ui, which is determined by the level
of income (x) that the individual possess. We further assume that Ui is
positively and linearly related to xi like:
U i = β 0 + β1 xi (5.24)

An important question now is: ‘how is the unobserved (latent) utility index
(Ui) related to the actual decision of owning a car’? We further assume that
there exists a cut-off or benchmark level of utility index, U i* , such that if the
actual utility index (Ui) exceeds the cut-off level of utility index U i* , the
individual will own a car. This means y i = 1, if Ui>Ui*, 0 otherwise. Given
the assumption of normality, the probability that U i* is less than or equal to
U i can be computed from the standardised normal CDF as:

pi p( y 1 / x) p (U i* Ui ) p ( zi 0 1 i x) F( 0 x)
1 i
(5.25)

where p ( y | x ) implies the probability of having a car given the income level,
z i is the standard normal variate (with zero mean and unit variance) and F is
the standard normal CDF, which can be written as:
Ii z2
1
F (U i ) e 2
dz
2

1 xi z2
1 0

e 2
dz (5.26)
2

97
Advanced Topic Since, p represents the probability that an event will occur (which is here the
in Regression
Analysis probability of owning a car), it is measured by the area of the standard normal
curve from − ∞ to U i . Now, from Equation (5.25), we can write:

U i = F −1 (U i ) = F −1 ( Pi ) = β 0 + β1 xi (5.27)

where F-1 is the inverse of normal CDF. Now, the data may be at the
individual or group level. The same process as was discussed in the Logit
model, needs to be applied here. The only difference is the CDF used. The
parameters of β 0 and β 1 are to be estimated from the ‘maximum likelihood
method’. The ‘marginal effect’ of Probit model is obtained by differentiating
the Equation (5.25) with respect to xi i.e.:
∂pi
= F / .β1 (5.28)
∂xi

where F/ is the ‘probability density function’ of the standard normal variate

z i = β 0 + β1 xi . F/is to be evaluated at zi.

5.5.1 Empirical Example of Probit Model

We again consider the earlier example of car ownership as in Table 5.3. The
estimated Probit model is reported in Table 5.5.

Table 5.5: Estimated Probit Model

Probit regression Number of observations=25

LR chi2(1) =21.53 Prob> chi2 =0.0000

Log likelihood = -6.5456442 Pseudo R2 =0.6218

Dependent Var Coefficient Std. Err. z P>z

Income 0.1620 0.05848 2.77 0.006

Constant -7.6841 2.782351 -2.76 0.006

The Probit coefficient is 0.1620. This means if income (x) increases, the
actual utility index (Ui) exceeds the threshold level of Utility Index (U*i)
with the probability of owning a car increasing by 16.2%. The intercept term
is negative. This means that if income is zero, the utility index is also
negative leading to ‘zero’ probability of owning a car. Now, the marginal
effect can be estimated for any value of the level of income. Earlier we saw
that there exists only one marginal effect of Logit model which is evaluated
at mean value. But, in case of Probit model, one can estimate the ‘marginal
effect’ for any positive value of x. This is done by estimating the probability
density of the standardised normal curve.

98
5.5.2 Estimation of Marginal Effect in Profit Model Binary Dependent
Variable Models
∂pi
Suppose we want to evaluate the marginal effect, at the mean value of x
∂xi
i.e. x = 47.2. Using, 5.27, we can compute the marginal effect of the Probit
model as:
pi
F /. 1 f ( 7.6841 0.1620 47.2) 0.1620 f ( 0.03) 0.1620
xi
0.4750 0.162 0.076

In the above we have taken the value 0.475 because the value of f (−0.03)
corresponding to P(–1.96 ≤ Z ≤ 0 ) is 0.475. The results of the ‘marginal
effect’ means that for one unit increase in income (x) beyond the mean
income ( x ) , the probability of owning a car will increase by 7.6 percent. This
is more or less same as the Logit result (in case of Logit, we estimated the
‘marginal effect’ as 0.073). One can estimate the ‘marginal effect’ for any
value of x in Probit model but we cannot do it in the Logit model. The Logit
and Probit models are more or less similar. The sign of the coefficients are
same. The only difference is related to numerical values. Empirically, one
can establish the relationship between the two coefficients of Logit and Probit
models by multiplying the Probit coefficient by 1.81. This gives us an
approximate value of the Logit coefficient. This can be examined in our
above car ownership example. The estimated Logit coefficient is 0.293 while
the estimated Probit coefficient is 0.1620. You can verify that 0.293≈1.81 ×
0.1620. Therefore, in general, we can write this as:

β1 of Logit ≈1.81 × β1 of Probit (5.29)

There also exists a relationship between the coefficients of Logit model and
LPM as follows:
β1 of LPM ≈ 0.25 × β1 of Logit [except for intercept] (5.30)

β1 of LPM ≈ 0.25 × β1 of Logit +0.5 [including intercept] (5.31)

You can verify the above relationship by estimating the LPM which is not
done here in this unit.

Check Your Progress 2 [answer within the space given in about 50-100
words]

1) Define the Logit model and state its properties.

.....................................................................................................................
.....................................................................................................................
.....................................................................................................................
.....................................................................................................................
.....................................................................................................................
99
Advanced Topic .....................................................................................................................
in Regression
Analysis .....................................................................................................................
2) In the Logit model, to estimate the ‘marginal effect’ which two
information are needed?
.....................................................................................................................
.....................................................................................................................
.....................................................................................................................
.....................................................................................................................
.....................................................................................................................
3) What is a fundamental difference between the Logit model and the Probit
model?
.....................................................................................................................
.....................................................................................................................
.....................................................................................................................
.....................................................................................................................
.....................................................................................................................
4) Empirically, what is the relationship that exists between: (i) the Logit
coefficients and the Probit coefficients and (ii) the Logit coefficients and
the LPM coefficients?
.....................................................................................................................
.....................................................................................................................
.....................................................................................................................
.....................................................................................................................
.....................................................................................................................
5) In Tables 5.4 and 5.5 we have used values of Z. But values of chi-square
is also given. Write a comment on this.
.....................................................................................................................
.....................................................................................................................
.....................................................................................................................
.....................................................................................................................
.....................................................................................................................

5.6 LET US SUM UP

Qualitative response regression model refers to models in which the response
for the endogenous variable is not quantitative on an interval scale. There are
100 three alternative models viz. the LPM, the Logit and the Probit to deal with
dummy dependent variable situations. The unit has explained these three Binary Dependent
Variable Models
models for situations of ‘binary dependent variable models’. The LPM is the
simplest one but it has serious limitations. The most important model is
Logistic regression or Logit model. The limitations of LPM are overcome
both in the Logit and the Probit models. There is no significant difference
between Logit and Probit models. Their regression coefficients are related.
This means, if we get a regression coefficient of Logit model, we can
indirectly estimate the Probit regression coefficients. The regression
coefficients of LPM and Logit model are also related. The unit has indicated
these relationships.

5.7 KEY WORDS

Linear : It is stated as in Equations (5.4) and (5.5) where

Probability dependent variable takes two values: 0 and 1. In this, we
Model have to deal with situations on non-normality of
disturbance terms, heteroscedastic variances of
disturbance terms and non-fulfilment of conditional
probability values being within the limits of ‘0’ and ‘1’.
Logit Model : This is a model as stated in Equation (5.10). This is non
linear both in x’s and β. Hence, the OLS method of
estimation does not work here. We can estimate the
parameters by using the ML method as outlined in
Equation (5.16). The test results of this model are based
on standard normal Z values.
Probit : The form of the Probit model depends on the ‘cumulative
Model distribution function’ (CDF). Usually, we work with the
‘normal variate’. Hence, this is defined as: the
econometric model based on ‘dummy dependent variable’
as emerges from the normal distribution’s CDF. As in the
Logit model, the test results of this model are also based
on the Z values.

5.8 SUGGESTED BOOKS FOR FURTHER

READING
1) Gujarati D., Porter D. C. and Gunasekhar S (2018). Basic Econometrics,
5th Edition, McGraw Hill Education (India) Pvt. Ltd.
2) G.S. Maddala,. (1992). Introduction to Econometrics, Second Edition,
Macmillan, New York.

101
Advanced Topic
in Regression
5.9 ANSWERS/HINTS TO CHECK YOUR
Analysis PROGRESS EXERCISES
Check Your Progress 1

1) The word ‘binary’ restricts the dependent variable to be dichotomous.

But among the independent variables we can have a normal variable like
income or a dummy variable or a mixture of both. This means the
regressors can be either quantitative or qualitative.
2) No. We encounter three specific type of problems viz. variances of
disturbances are not homoscedastic, the probability values of conditional
expectation is not between ‘0’ and ‘1’ and the error terms are not
normally distributed, a property that we do not need for OLS but we
need for drawing ‘statistical inferences’.
3) We can apply the method of weighted least squares (WLS) by taking
∧ ∧ ∧
wi y i (1 − y i ) as the estimate of wi .
=

4) This is because the conventional R 2 is likely to be much lower than 1 for

such models.

Check Your Progress 2

1) A Logit model is stated as in Equation (5.10). We cannot straightaway

apply the OLS method because p i is nonlinear not only in x but also in
the β’s. But by considering the ratio as in Equation (5.12) and then taking
logarithm on both sides of this equation, we get an equation like (5.13)
which is linear in xi. For estimation purposes, we have to work with Li as
stated in Equation (5.15). If data is available at individual level, we
cannot estimate its parameters by the OLS method. For this reason, we
take the help of the ML method to solve for its parameters.
2) (i) estimated Logit model and (ii) the mean value of the regressor.
3) One can estimate the ‘marginal effect’ for any value of x in Probit model
but we cannot do it in the Logit model.
4) The relationships are approximations as can be obtained by Equations
(5.29) to (5.31).
5) The likelihood-ratio (LR) test assesses the ‘goodness of fit’ of two
competing statistical models based on the ratio of their likelihoods.
Specifically, one is found by maximisation (over the entire parameter
space) while another is found after imposing some constraint. If the
constraint (i.e. the null hypothesis) is supported by the observed data, the
two likelihoods should not differ by more than the sampling error. Thus,
the likelihood-ratio test tests whether this ratio is significantly
different from one, or equivalently, whether its natural logarithm is
significantly different from zero. If the distribution of the likelihood ratio
102 corresponding to a particular null (and the alternative hypothesis) can be
explicitly determined, then it can directly be used to form decision Binary Dependent
Variable Models
regions (i.e. to sustain or reject the null hypothesis). In most cases,
however, the exact distribution of the likelihood ratio corresponding to
specific hypotheses is very difficult to determine. However, following
Wilk’s theorem, we can argue that as n (sample size) tends to infinity,
the test statistic asymptotically will follow the Chi Square distribution.

103