142024 Optimization Techniques popularly used in Deep Leaming | by Aart Kasirajan | Medium Openin app Aarthi Kasirajan 16 Followers About You have 2 free member-only stories left this month. Sign up for Medium and get an extra one Optimization Techniques popularly used in Deep Learning @ Aarthi Kasirajan Sep 9, 2020 - 7minread * [Link]/@minions Kloptimization-techniques-popularly-usedn-deop-oarning-3c21BecBece am142024 Optimization Techniques popularly used in Deep Leaming | by Aart Kasirajan | Medium In this article we will be looking briefly at various optimization techniques widely used in Deep Learning. Before reading this article, if you wanted the mathematical explanation and equation, then this is not for you. This article simply briefs about the various techniques available and the ones that Keras package gives us. Insimple words, Optimization algorithms are responsible for reducing losses and provide most accurate results possible. The weight is initialized using some initialization strategies and is updated with each epoch according to the equation. The best results are achieved using some optimization strategies or algorithms called Optimizer. ‘atch gradient descent (Gradient calevated ‘on enire data se) Schatten coon aus otsttad tr ocppemion uetamen per epost) ‘ NAG Reduce ner by sng ep) watch gradont descert (Simi to SCD but en mini batches. Note Thsis atten Momentum Trotng tbl donna -Gotaser mioredieae Seu) ‘engi a creer Seagaoanps aring sn om pamesaae ‘rotors races it has earn ae sro a S esr RSPop + tar-conecn + name) Mind Map of Optimizer Some of the techniques that we will be discussing in this article is- + Gradient Descent + Stochastic Gradient Descent (SGD) + Mini-Batch Stochastic Gradient Descent (MB — SGD) [Link]/@minions Kloptimization-techniques-popularly-usudn-deop-earning-3c21BecBedce amt142024 Optimization Techniques popularly used in Deep Leaming | by Aart Kasirajan | Medium + SGD with Momentum + Nesterov Accelerated Gradient (NAG) + Adaptive Gradient (AdaGrad) + AdaDelta - RMSProp + Adam +Nadam 1. Gradient Descent *b Ws x Gradient Descent algorithm ‘A Gradient Descent is an iterative algorithm, that starts from a random point on the function and traverses down its slope in steps until it reaches lowest point of that function. This algorithm is apt for cases where optimal points cannot be found by equating the slope of the function to 0. For the function to reach minimum value, the weights should be altered. With the help of back propagation, loss is transferred from [Link]/@minions Kloptimization-techniques-popularly-usedn-deop-oarning-3c21BecBece sitvazo2s Optimization Tecniques pop sed in Deep Learing | by Aart Kasrajn | Medium one layer to another and “weights” parameter are also modified depending on loss so that loss can be minimized. Cost function: 0=0—a-VJ(0) 6F T T T aie global maximum local maximum ab 0 2 - local minimum “4 cai global minimum C | L | | | As for Gradient Descent algorithm, the entire data set is loaded at a time. This makes it computationally intensive. Another drawback is there are chances the iteration values may get stuck at local minima or saddle point and never converge to minima. To obtain the best solution, the must reach global minima. 2. Stochastic Gradient Descent (SGD) Stochastic Gradient Descent is an extension of Gradient Descent, where it overcomes some of the disadvantages of Gradient Descent algorithm. SGD tries to overcome the disadvantage of computationally intensive by computing the derivative of one point at a [Link]/@minions Kloptimization-techniques-popularly-usedn-deop-oarning-3c21BecBece ants1a021 Coptmization Techniques popuaty used in Deep Learning| by Aart Kasrajan | Mesum time. Due to this fact, SGD takes more number of iterations compared to GD to reach minimum and also contains some noise when compared to Gradient Descent. As SGD computes derivatives of only 1 point at a time, the time taken to complete one epoch is large compared to Gradient Descent algorithm. 3. Mini Batch — Stochastic Gradient Descent MB-SGD is an extension of SGD algorithm. It overcomes the time-consuming complexity of SGD by taking a batch of points / subset of points from dataset to compute derivative. Note- It is observed that the derivative of loss function of MB-SGD is similar to the loss function of GD after some iterations. But the number iterations to achieve minima in MB- SGD is large compared to GD and is computationally expensive. The update of weights in much noisier because the derivative is not always towards minima. In recent times Adaptive Optimization Algorithms are gaining popularity due to their ability to converge swiftly. These algorithms use statistics from previous iterations to. speed up the process of convergence. In the next segment of the article we will be looking at some Adaptive Optimization Algorithm techniques. rere OPTIMIZATIO TECHNIQUES 4. Momentum based Optimizer [Link]/@minions Kloptimization-techniques-popularly-usedn-deop-oarning-3c21BecBece sts1a021 Coptmization Techniques popuaty used in Deep Learning| by Aart Kasrajan | Mesum It is an adaptive optimization algorithm which exponentially uses weighted average gradients over previous iterations to stabilize the convergence, resulting in quicker optimization. This is done by adding a fraction (gamma) to the previous iteration values. Essentially the momentum term increase when the gradient points are in the same directions and reduce when gradients fluctuate. As a result, the value of loss function converges faster than expected. Cost function: V(t)=yV(t—1) +a. VJ(O) 0=0-V() —*x 11.54 — beta = 0.4 — beta = 0.6 15.0 | — beta = 0.9 — beta = 0.99 RS 10.0 15 5.0 25 0.0 0 10 20 30 40 For different values of B, the performance of algorithm is plotted 5. Nesterov Accelerated Gradient (NAG) In momentum-based optimization, the current gradient takes the next step based on previous iteration values. But we need a much smarter algorithm that knows when to intuitively stop so that the gradient doesn't further increase. ‘To do this the algorithm should have an approximate idea of the parameter values in its next iteration. In doing so we can efficiently look ahead by calculating the gradient values wrt to future position of parameters. [Link]/@minions Kloptimization-techniques-popularly-usedn-deop-oarning-3c21BecBece ents1a021 Coptmization Techniques popuaty used in Deep Learning| by Aart Kasrajan | Mesum From the previous equation, we know that momentum includes the term [yV(t—1)] to calculate the value of previous iterations. Computing (0-yV(t—1)) gives us an approximation of next position of parameters[@]. We can now conclusively look ahead of current parameters by approximating future position with the help of below equation. Cost function: V(t)=yV(t—1) +a. VJ[0-yV(t- D)] @=0—Vi) By using NAG technique, we are now able to adapt error function with the help of previous and future values and thus eventually speed up the convergence. Now, in the next techniques we will try to adapt alter or vary the individual parameters depending on the importance factor it plays in each case. 6. AdaGrad Adaptive Gradient as the name suggests adopts the learning rate of parameters by updating it at each iteration depending on the position it is present, i.e- by adapting slower learning rates when features are occurring frequently and adapting higher learning rate when features are infrequent. Technically it acts on learning rate parameter by dividing the learning rate by the square root of gamma, which is the summation of all gradients squared. In the update rule, AdaGrad modifies the general learning rate N at each step for all the parameters based on past computations. One of the biggest disadvantages is the accumulation of squared gradients in the denominator. Since every added term is positive, the accumulated sum keeps growing during the training. This makes the learning rate to shrink and eventually become small. This method is not very sensitive to master step size and also converges faster. 7. AdaDelta It is simply an extension of AdaGrad that seeks to reduce its monotonically decreasing learning rate. Instead of summing all the past gradients, AdaDelta restricts the no. of summation values to a limit (w). In AdaDelta, the sum of past gradients (w) is defined as “Decaying Average of all past squared gradients”. The current average at the iteration then depends only on the previous average and current gradient. [Link]/@minions Kloptimization-techniques-popularly-usedn-deop-oarning-3c21BecBece mms1a021 Coptmization Techniques popuaty used in Deep Learning| by Aart Kasrajan | Mesum 8. RMSProp Root Mean Squared Prop is another adaptive learning rate method that tries to improve AdaGrad. Instead of taking cumulative sum of squared gradients like in AdaGrad, we take the exponential moving average. The first step in both AdaGrad and RMSProp is identical. RMSProp simply divides learning rate by an exponentially decaying average. 9, Adaptive Moment Estimation (Adam) It is a combination of RMSProp and Momentum. This method computes adaptive learning rate for each parameter. In addition to storing the previous decaying average of squared gradients, it also holds the average of past gradient similar to Momentum. Thus, Adam behaves like a heavy ball with friction which prefers flat minima in error surface. Wr =(1—A)w,— VF. (w,) weight decay update Momentum update Nesterov momentum update “lookahead” gradient stop (bit different than momentum momentum original) step step actual step actual step gradient step sourec: cs231n lecture notes. 10. Nesterov Accelerated Adaptive Moment Estimator (Nadam) As we have seen in previous section, Adam is a combination of RMSProp and Momentum. Earlier we have also seen that NAG is superior to momentum. Nadam thus incorporates Adam and NAG. To do this we need to modify the momentum term. oan Tesi leas wow Validation loss “el coe ee 08 sean SEs. tps imei. com/@minions Kloptimization-techniques-popularly-usedn-deop-loarning-3c2BecBedce ant142024 Optimization Techniques popularly used in Deep Learning by Aath Kasiajan | Medium | NSS |B tan SS wos SSS : tors : ay Pat : (05 i ay ws a Epoch, Epoch NSE source: Nadam paper SGD Momentum NAG Adagrad Adadelta Rmsprop ‘Though we have discussed some types of Optimization techniques, not all of these are provided by the popular Keras package. The methods provided by Keras are as listed below- SGD BMSprop Adam [Link]/@minions Kloptimization-techniques-popularly-usudn-deop-earning-3c21BecBedce ont142024 Optimization Techniques popularly used in Deep Learning | by Aath Kasiajan | Medium Adadelta Adagrad Adamax Nadam Fel To get in detail explanation on how to implement in you Deep Learning model, click on the particular link. Now, that we have had a look at the different optimization techniques, we cannot implement all of them for a same problem. Depending on the case of the problem the approach may change. Now, its your time to decide which optimization technique you want to use in your model. Conclusion Adam is definitely one of the best optimization algorithms for deep learning and its popularity is growing very fast. While people have noticed some problems with using [Link]/@minions Kloptimization-techniques-popularly-usedn-deop-oarning-3c21BecBece som61412021 Optimization Techniques popularly used in Deep Leaming | by Aarthi Kasirajan | Medium Adam in certain areas, researches continue to work on solutions to bring Adam results to be on par with SGD with momentum. APPY ARNIN G Deep Learning Optimization Algorithms Data Science [Link]@minions Kloptimization-echniques-popularly-usedn-deop- wt