Logistic Regression and machine learning :

Logistic regression is a binomial regression model. As with all binomial regression models,
the aim is to best model a simple mathematical model with many real observations. In other
words to associate a vector of random variables ( x1, x2, …, xk) with a binomial random
variable generically denoted y. Logistic regression is a special case of a generalized linear
model and it is widely used in machine [Link] brings curiosity to understand more the
use of this model in traditional statistics but more specifically in machine learning. So what
are the advantages of this model compared for example to the linear regression ? When
could we use this model and what is the use of the logistic regression in Finance ?

In recent years, computer science and programming are rapidly emerging as fields of study.
With the computerization of companies, the data collected is more and more numerous and
the questions of storage and manipulation of this data are puzzles for mathematicians and
computer scientists. This is what gave birth to the very general term of Big Data.
Several techniques from statistics and probability have increased knowledge about data
analysis, deleting outliers or managing missing data to choose a relevant representation of a
phenomenon. Once the data is well prepared, the question arises of how to derive effective
information from large quantities of data that would require procedures that are too greedy in
computing resources and underdeveloped knowledge.

This is where machine learning comes in, making a program capable of learning from
example data without being programmed.

The idea is easy to understand, it is similar to the learning of a human being who after
several experiments on the same subject becomes more and more efficient and
autonomous.
All machine learning models are classified into two categories: supervised or unsupervised.
If the model is a supervised model, it can be of 2 types or subcategories: regression or
classification model.

Supervised learning involves teaching a function to match an input to an output based on

For example, if we had a dataset with two variables, age (input) and height (output), we
could implement a supervised learning model to predict a person's height based on his age.
In classification models, the output is discrete. And logistic regression is one of the most
common types of classification models.
Logistic regression and linear regression belong to the same family of GLM models
(Generalized Linear Models): in both cases an event is linked to a linear combination of
explanatory variables. For linear regression, the dependent variable does not follow a normal
distribution N (µ, s) where µ is a linear function of the explanatory variables. For logistic
regression, the dependent variable, also called the response variable, follows a Bernoulli law
of parameter p (p the average probability for the event to occur), when the experiment is
repeated once, or a Binomial law ( n, p) if the experiment is repeated n times (for example
the same dose is tried on n insects). The probability parameter p is here a function of a
linear combination of the explanatory variables.

The most commonly used functions to relate the probability p to the explanatory variables
are the logistic function and the distribution function of the standard normal distribution.

“When the variable to be explained is a qualitative variable, the classical linear regression at
least squares sense should be abandoned in favor of logistic regression.
If the variable to be explained has only two modalities, we use logistic regression
binary. If it has more than two modalities and if they are not ordered, we must
use polynomial logistic regression. Finally, if the variable to be explained has more
of two modalities and that these are ordered, the method to be exploited is the regression
ordinal”. (Adeline GILLET, 2010)

Logistic regression is a technique that can be used for traditional statistics as well as
machine learning. But before we dove into a logistic regression. Let's understand more linear
regression.

Let’s take for example data as the weight and size of a mouse. Then we fit a line to it. And
with that line, we could do a lot of things. First, we could calculate our squared and
determine if weight and size are correlated, large values imply a large effect. And second,
calculate a p value to determine if the R squared value is statistically significant. And third,
we could use the line to predict size given weight.

If a new mouse has this [Link] this is the size that we predict from the weight.

Using data to predict something falls under the category of machine learning. So plain old
linear regression is a form of machine learning. Another statistical technique of prediction is
the multiple regression. With this technique we could predict size using weight and blood
volume. Alternatively, we could say that we are trying to model size using weight and blood
volume. Multiple regression did the same things that normal regression did.

We can calculate R-squared and the p value. And we could predict size using weight and
blood volume. And this makes multiple regression, a slightly fancier machine learning
method. Also we can use discrete measurements like genotype to predict size. Lastly, we
could compare models. And we can compare those predictions to the ones we get from
multiple regression where we're using weight and blood volume to predict size. Comparing
the simple model to the complicated one tells us if we need to measure weight and blood
volume to accurately predict size or if we can get away with just weight.

Logistic regression is similar to linear regression, except. Logistic regression predicts

Logistic Regression could tell us if the mice are obese or not. Also, instead of fitting a line to
the data, logistic regression fits an S shaped logistic function. The curve goes from zero to
one. And that means that the curve tells you the probability that a mouse is obese based on
its weight.

Although logistic regression tells the probability that a mouse is obese or not, it's usually
used for classification. For example, if the probability of mouse is obese is greater than 50
percent, then we'll classify it as obese. Otherwise we'll classify it as not obese. Just like with
linear regression, we can make simple models in this case, we can have obesity predicted
by weight. Or more complicated models in this case, obesity is predicted by weight and
genotype.

In this case, obesity is predicted by weight and genotype and age. And lastly, obesity is
predicted by weight, genotype, age and astrological sign. In other words, just like linear
regression, logistic regression can work with continuous data like weight and age and
discrete data like genotype and astrological sign. We can also test to see if each variable is
useful for predicting obesity. However, unlike normal regression, we can't easily compare the
complicated model to the simple model.

Instead, we just test to see if a variable's effect on the prediction is significantly different from
zero. If not, it means that the variable is not helping the prediction. We use Wald's test to
figure this out. In this case, the astrological sign is totally [Link] means we can save
time and space in our study by leaving it out. Logistic regressions ability to provide
probabilities and classify new samples using continuous and discrete measurements makes
it a popular machine learning method.

One big difference between linear regression and logistic regression is how the line is fit to
the data. With linear regression, we fit the line using least squares. In other words, we find
the line that minimizes the sum of the squares of these residuals. We also use the residuals
to calculate R squared and to compare simple models to complicated models. Logistic
regression doesn't have the same concept of a residual, so it can't use least squares and it
can't calculate R-squared. Instead, the model uses something called maximum likelihood.

In summary, logistic regression can be used to classify samples. And it can use different
types of data like size and or genotype to do that classification. And it can also be used to
assess what variables are useful for classifying samples, i.e., astrological signs are totally
useless.

Classification is a very common task in machine learning. In this type of problem, we try to
put a label on an observation: is the mouse obese or not, is a tumor malignant or not, is a
transaction fraudulent or not… these two cases are examples of classification.
We used the example of a mouse to simply explain the use of the logistic regression
compared to other techniques like linear regression and multiple regression. But the use of
this technique doesn’t only concern biology or animal purpose, it goes beyond that and
Finance is another field to explore.

In fact the most common use of the logistic regression model in finance is estimating the
credit risk or the market risk.
For the Credit risk we can take the example of Credit Scoring, that could be estimated by a
Logistic regression model, and compared to other regression techniques, in particular linear
regression, logistic regression is essentially distinguished by the fact that the variable to be
explained is discrete (categorical). The ease of use of logistic regression is based on the fact
that it is not based on restrictive assumptions, one can go directly to the regression without
performing collinearity and homosedasticity tests.

Logistic regression is a frequent technique in credit scoring for solving binary classification
problems. Before fitting the model, it is useful to reiterate the variable selection to verify that
the transformed variables still fit the model. Preferred potential variables are those with a
higher informational value (usually between 0.1 and 0.5), which have a linear relationship
with the dependent variable, cover all categories, have a normal distribution, contain a
contribution of together meaning, and are applicable to the enterprise.

“Many analysis service providers include the logistic regression model in their software
products, typically with a wide range of statistical and graphing functions. For example, the
implementation of the SAS language LOGISTIC procedure in WPS provides a full range of
options for selecting variables, restricting model parameters, weighting variables, obtaining
separate analyzes for different segments, scoring on another dataset, generating automated
deployment code, for example”.

To conclude, logistics could help us determine (classify) a binary diagnosis. This algorithm is
also widely used in biology and finance.
What is very interesting in logistic regression and more generally in machine learning models
is that the algorithm does not need to know any theoretical rules linking the input of the
model and its output. The machine learning model learns these rules on its own based only
on data.
We can understand that logistic regression has become an important tool in the discipline of
machine learning. This approach makes it possible to use an algorithm in the machine
learning application to classify the incoming data based on historical data. The more relevant
data input, the better able the algorithm is to predict classifications within datasets.
Logistic regression can also play a role in data preparation, by allowing datasets to be
divided into specially defined categories during the process of extract, transform and load in
order to organize information for analysis.