Scribd
Upload
2 views8 pages

Deep Learning - 2

The document discusses the behavior of a linear model's function in relation to variables h1 and h2, highlighting how the function increases with h1 and decreases with h2. It explains the derivation of loss functions from Maximum Likelihood Estimation (MLE), noting that while linear models yield convex surfaces, non-linear models like neural networks result in non-convex surfaces due to non-linear activation functions. Additionally, it addresses properties of cross-entropy cost in MLE, emphasizing its lack of a minimum value and its implications for discrete and real-valued output variables.

Uploaded by

23ceubs023
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
2 views8 pages

Deep Learning - 2

The document discusses the behavior of a linear model's function in relation to variables h1 and h2, highlighting how the function increases with h1 and decreases with h2. It explains the derivation of loss functions from Maximum Likelihood Estimation (MLE), noting that while linear models yield convex surfaces, non-linear models like neural networks result in non-convex surfaces due to non-linear activation functions. Additionally, it addresses properties of cross-entropy cost in MLE, emphasizing its lack of a minimum value and its implications for discrete and real-valued output variables.

Uploaded by

23ceubs023
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd

Deep Learning -2

• The linear model can now describe the function as increasing in h1


and decreasing in h2.

• How ? The partial derivative of the function z = h1-2h2. It is increasing


on h1 and decreasing on h2. This behavior can also be observed from
the dataset ; the function value increases from 0 to 1 for constant
value of h2 and increasing the value of h1. While the function
decreases with increasing the value of h2.
• Initially the loss functions are derived from the Maximum Likelihood
estimate. For linear functions the loss function that was derived: MSE
was having a convex surface.

• While when the same MLE is used for non-linear models such as
Neural network, the MLE becomes a function of non-linear activation
functions and so the surface also becomes non-convex surface.
Hence, the non-linearity drives the loss function to become non-
convex.
Maximum likelihood estimation under a
Gaussian model
• We are modelling:

• This means:
• Given input x, your model predicts a mean f(x;θ)
• You assume Gaussian noise with:
• Mean: f(x;θ)
• Covariance: Identity matrix I
Cross Entropy and Regularization
• A property of cross-entropy cost used for MLE is that it does not have
a minimum value
• – For discrete output variables, they cannot represent
• probability of zero or one but come arbitrarily close
• • Logistic Regression is an example
• – For real-valued output variables it becomes
• possible to assign extremely high density to correct training set outputs, e.g,
by learning variance parameter of Gaussian output, and the resulting cross-
entropy approaches negative infinity

You might also like