Neural Networks and Deep Learning (CSE 4985)Sahoo An Introduction to Neural Networks 1 Introduction 1. Artificial neural networks (ANN) are popular machine learning techniques that simulate the mechanism of learning in biological organisms. n contains cells called neurons that are con ig 2. The human nervous syste nected to one another usi axons and dendrites as represented 3. The connecting regions between axons and dendrites are referred to as synapses. A simple neuronal structure in hnman nervous system is as represented in Fig.2 Figure 2: A simple neuron 4. The basic building block of every artificial neural network is an artificial neuron that is represented with a simple mathematical function or model. ‘The computation units in artificial neural networks are called as neurons. 5. Such a model will have three simple sets of rules: multiplication, summa- tion and activation, 6. At the entrance of every artificial neuron, the inputs are weighted means, they are multiplied with their corresponding weights. Scanned with CamScanner scanned with Camscanner7. At the end of every artificial neuron, the sum of previously weighted inputs ‘and a bias is passed through an activation function for computing the target variable, 8, Hence an ANN computes a function of inputs by propagating the com- puted values from input neurons to output neurons using the weights as the intermediate parameters 9. Learning occurs by changing the weights connecting the neurons. The taining data with input-output pairs help the neural funetion to learn and update itself, 10. This ability of accurately computing functions of unseen inputs by training over a finite set of input-output pairs is called as model generalization. Limits of artificial intelligence ARTIFICIAL INTELLIGENCE Figure 3: Cambrian Explosion in deep learning. 1. The broadest concept AI is the science of making machines perform tasks that usually require human intelligence. AI is the "umbrella term” under which ML and Deep Learning fall. Scanned with CamScanner scanned with Camscanner2. Machine learning is a subset of AI that focuses on teaching machines to learn from data and improve performance without being explicitly pro- grammed. Its goal is to extract patterns from data and make predic tions /decisions. 3. Decp learning is a subsct of Machine learning that uses artificial neural networks with many layers (hence “deep”). Its goal is to learn complex patterns and representations automatically often from unstructured data {like images, audio, text). Hence DL isa subset of ML and ML isa subset of AL and this is represented in Fig 4. The performance of traditional machine learning remains better at times for smaller data sets because of more choices, greater ease of model i terpretation and the tendency to interpret features incorporating, domai specific insights as represented in Fig.3. Figure 4: ML and DL: Subsets of AL. ‘The performance becomes stagnant in ML after a certain period even with. increase in the size of the data as represented in Fig.4 whereas the Deep learning performance goes cn increasing wit the available data. the increase in the size of 2.1 The Basic idea of Neural Networks 1. A neural network is structured as a directed acyclic graph where the net work edges are parameterized with weights such that the function com- puted at individual nodes are affected by the weights of incoming edges ‘as well as variables in the nodes at the tails of these edges. Scanned with CamScanner scanned with Camscanner2. The overall function computed by a network isthe result of easeading fune- tion computations at individual nodes. Always the neurons are arranged in a layered fashion inside the network, 3, Structure of a Basic Neural Network: (a) Input Layer: Takes raw data (features).Example: For an image, each pinel isan input. () Hidden Layer(s):Neurons here apply mathematical fetions to trans- form input into something useful. Uses weights, biases, and activation functions. (c) Output Layer:Produces the final prediction or classification. Exam- ple: For spam detection — ontputs "Span" ot "Not Spam" (d) Key Components:Neuron (Perceptron): Each neuron calculates: peur twa tre tua aati th a where w = weights, x = inputs, b = bias (c) Activation Function:Introduces non-linearity, so the network ean solve ‘complex problems, Common ones:Sigmoid, ReLU (Rectified Linear Unit), Tanh, (€) Forward Propagation: Data flows from input + hidden -» output. (g) Loss Function: Measures how far the prediction is from the correct Example: Mean Squared Error, Cros-Entropy. (h) Backpropagation: Adjusts weights & biases by calculating gradients (errors) and updating with Gradient Descent. 3. Single computational layer: The perceptron in, + bias Figure 5: A simple perceptron. Scanned with CamScanner scanned with Camscanner‘What is a Perceptron (a) A perceptron as in Fig.5 is a single computational layer (a single neuron) introduced by Frank Rosenblatt (1958). It's the simplest type of neural network. ‘and can solve only linearly separable problems. (b) A perceptron takes inputs, multiplies them by weights, adds a bias, and applies an activation function to decide the output. 1. STRUCTURE AND FUNCTIONS OF ARTIFICIAL NEURO? An artificial neuron is a mathematical function conceived as a model of biological neurons, a neural network, Artificial neurons are elementary units in an artificial neural network. ‘The artificial neuron receives one or more inputs (representing excitatory. postsynaptic potentials and inhibitory postsynaptic potentials at neural dendrites) and sums them to produce an output (or activation, represent- {ng a nenton’s action potential which is transmitted along its axon) Usually each input is separately weighted, and the sum is passed through ‘non-linear function known as an activation function or transfer function, ‘The transfer funetions usually have a sigmoid shape, but they may also take the form of other non-linear functions, piecewise linear functions, or step functions They are also often monotonically increasing, continuous, differentiable ‘and bounded. 2. Mathematical basics of an ANN: Basic ANN architecture with and without bias is as represented in Fig.6 Figure 6: Pereeptron without bias Figure 7: Perceptron with bias and Fig. Scanned with CamScanner scanned with CamscannerAs the training data has d inputs denoted by row vector bars: and a single output y. Hence input layer contains d nodes that transmit d features, 274..t4 contained in row veetor X. ‘The input layer is incident edges of weight 1)..g contained in column vector WW, The input layer does not perform any computation on its own, ‘The linear function RT =) wins @ is computed at the output node. The computed output jis the predicted value that is ealculated as follows: w F) sign\V.R™ = sign} wx, ® ‘The sign function maps real value to either +1 of -1 that is appropriate for binary classification, ‘The difference between the the actual output and the predicted output is the loss function that is used to quantify positive eost when yis diferent from ‘The weights of the neural network are learned sregate loss over all training instances. Ini and are randomly initialized, order to minimize the ag- ly the weights are unknown Hence j is random and will never match y and the goal of learning pro- cess is Lo use these errors to modify the weights such that neural network: predictions will become more accurate. The learning happens according to the available training data. ‘The perceptron architecture is as shown in fig.6 where a single layer trans- ints the features to the output node. The network edges are parameterized with weights to which the corresponding features are multiplied and added at the output node. Subsequently sign function is applied to couvert the aggregate value into a class Inbel ‘The sign function serves as an activation function and the value of the variable in a noural network node is called as activation. A pereeptron’s input layer does not perform any computation and only ‘transmits the feature values. This is true across all neural architectures as input layer is only working as a transmission layer. [As the perceptron contains a single active layer, hence itis called as si ale computational layer or a single layered network. ‘The algorithm is Scanned with CamScanner scanned with CamscannerSahoo proposed by Rosenblatt where the weights are learned using a heuristic update process. Computation oy 42 + wy 4p ty #25 to +b “ Where: 2; -+ input features ‘uy > weights (Importance of each input) b— bias (shifts the decision boundary) f(z) + activation funetion (e.g, step, sigmoid, ReLU) 3.1 Use of bias in a neural network 1. What is Bias? Bias is an extra parameter added to a neuron, just like weights, but it is not multiplied by an input. . Why Do We Need Bias? It Shifts the Activation Function, Without bias, the output always passes through the origin (0,0). Bias lets the network shift the decision boundary left and right or up and down, Example: Like the “intercept” in a linear equatio Dias, sche, where © is It also increases Flexibility. With only weights, neurons can learn limited fun terns. fons. Adding bias allows the network to model a wider range of pat- It prevents Zero Output Traps. If all inputs are zero, the output would always be zero (without bias). Bias ensures the neuron can still fie (acti- vate) 4 Which objective function is the perceptron optimizing? Moat machine learning algorithms are loss optimization problems where gradi- et descent update are wed to minimize the lo. The basic neural architecture is motivated by Mc Culloch Pitts model of the neuron rather than the basic tmethomatical model. The primary gel i t minimise the aber of elastics the heuristic update process so that weights can be changed in Ue conte lection whenoaver‘ermcs pare mas tion errors with Scanned with CamScanner scanned with CamscannerSubhahuxmi Sahoo Conse ining instance (Ky), where Xi ow veto containing the fates and yc the olnerved clive variable Tas pereptronererion [lx mw fnetion that enalins the training instance when the sign of T= i diferent from = may TE), 0) as) Furthermore, lange abedte alas of WT that al do not match the sign of ys ae ‘enalza to a preatr de Gradient dent petra updater of Tt the nogatve ‘Sewetion ofthe radon of Ly (with expect wo TT) One en ells to ey that the ‘dient the be a ows Be] “Yo cherie 22 (o0 sg then Wes he tha ti ip nn he reapect 10 « ctor, ad simply the d-dimensonal vector [24... gle", The negative Sf cr ban oh ta mieten Samat blayomeeonseten Wew-ail, W + ay.X1 (For misclaited instances) ‘ham faction exp me of the wenn f the pt in the xg! lw Sear W tol any veoe tray of he tel dig st rend to ten Fal bs of In oa of Ofc, he yor epee conf cee ‘Tearing natin Set ee ttn sven tems nne i pv oom tan tat Drom wre caer ects ehh tng W-section oy rch ond ery ning tenes a). omc: he nto te ppm spat fr at a ro at pee irate ean ancl the ing oe sorties wt even ood eprint prt f incon Te wrt popriy of ie map ots wag ‘wreath goa das peecon by ety etcng be mapas File improving the nx Became of thi fc, the (alla) peoeptom in tot stable and an yd solutions of widely varying quality (fr iseparae cases). Several variations of Ue perceptron wore there proposed for ieparable dat, and natural appro i sways lop track of the best solution in terms of the umber of misctasfiatons [123 ‘This approach of sways Ieeping the best solution in one pocket” i refered to as the pocket algorithm. 5 Basic components of a neural architecture ‘The basic components ofa neural architecture are; Input layer, Hidden lagers, Output layer, Weights, Bias, Activation functions, Lows function, Optimizer, and Back propagation, 1. Input Layer: Accepts the raw data (features). Each neuron = one feature, Example: For an image (28x28 pixels), there are 784 input neurons. 2. Hidden Layers:Intermediate layers between input and output. Bach hid- den neuron: seunear tunes busses tot 6) Scanned with CamScanner scanned with Camscanner‘Then 2 passes through an activation function (e.g., ReLU, Sigmoid). Hid- den layers allow the network to learn complex patterns. 3. Output Layer:Produces the final result (classification, regression, ete), Example: Spam detection -> 2 outputs (Spam / Not Spam) Digit recos- nition (MNIST) + 10 outputs (digits 0-9) 4. Weights (w):Numbers that determine the importance of each input. These are learned during training and are ddjusted using gradient descent + backpropagation, 5. Bias (b):Extra trainable parameter that allows shifting of the activation funetion, Works like the intercept in linear equations. 6. Activation Functions: Introduce non-linearity so the network can model complex data. Common ones: Sigmoid — squashes output to (0,1) ReLU — keeps positive values, sets negatives to 0 ‘Tanh + squashes to (-1,1) 7. Loss Funetion:Measures how wrong the network becomes and this loss fimetion is reverse engineered to caleulate the neuron weights. Examples ‘Mean Squared Error (MSE) -» regression (Cross-Entropy > elasification 8. Optimizer: Algorithm that updates weights and biases to reduce loss. Ex- amples: Gradient Deseent,Adam, RMSProp 9. Backpropagation: Itis the leaning process that uses gradients (derivatives) to adjust weights and biases, It repeats until the loss is minimized. 6 Multilayer neural networks Unlike perceptron that contains a single input layer and an output layer, these ‘multilayer neural networks (MLL) contains more than one computational layers ‘These multiple additional computational layers are called as the hidden layers whose computations are not visible to the user. ‘The specific architecture of MLL are referred to as feed-forward networks because the successive layers feed into one another in the forward direction from input to output. ‘The default architecture assumes that all nodes in one layer are connected to those of the next layer. ‘The loss function penalizes undesirable derivations of predicted outputs of neural networks from observed outpus in the training data, Similar to case of single layer networks, bias neurons ean be used both in the hidden layers and Scanned with CamScanner scanned with CamscannerSubhaluxmi Sahoo in the output layers. ‘The diagram representing an MLL is as shown in the following fig. 8. 10 Scanned with CamScanner Scanned with CamscannerSubhahuxmi Sahoo 1.8.1 Choice of Activation Function ‘Thectiice of activation function i eral part of neural network dein. Dillereat ype talento such se Ue nigh sigmotd or hyertoke tangents ar cmon ted. ‘We have ately son the eof sgn ation inthe pecepton- We ws the notation to denote the activation funtion. A singe-hyer network with column vector of weighs tnd input (re vector X woul! have a prediction ofthe flowing form a= 40x") a6) ‘The most basic activation fanetion 9) thelr activation, which x ao refered oa the ety activation Hone ‘The lina activation function is fen wed in the output node, when the target & a real vale "The cla activation fictions that wore wad cary in the develope of eur setworks were a gn sated. he yer ange ati Fig 1.6 Pro and yt tition a within a eo ‘The break-up of the rerun computations into two separate asi shown in Figure 1.6.4 testo all compat oon thin te bic nly hae iterate {Usrumntion symbol Ua el ae he eatin end @ within tot, The ae ome puted ble apg the cain ton) wl be fered to Ue preacteation fate, wee the tle compte api be ciation tom bere Wo a the postacination value A ipa ol that erg res Figure 16 aoe ‘cul tat we with sie tian nn pre patio edo ae ‘thik pti the incr tart ag = Wal hee pers the niet {rats () Ul th 9p of gel trate en el ee to ‘sp arin tp altel eral ate u Scanned with CamScanner scanned with CamscannerSubhahuxmi Sahoo ian moni eee ew fr creaing hn fet at tg te Foe ‘ample wt he prereset sign fio or pict, hppa rein oly nar ataton The ig action tp ado (01). whch ea fn iaerpretng oats uproot ftom ha hap mart thao the sigmoid neti, exe that ite briny stretch and wera rama omclec {o[-th1) Te tanh a sig uct ae related els awe Exerc 3) tanh(o) = 2- sigma 20)—| ‘The tah function ie prrle o the sgncid when the outputs ofthe computations are dete to be both ptive and negative: The Samo and the tanh factors have be the straw choice for incorpcrating nonin i he neal ewok. fn cet svar howe a ero pecrnte er nation hase te ts Ppa (e) = maro,0}(Rectiel Liner Ut [RLU (0) = mn info, =1) lr tan) ‘The RALU and hard tanh activation fein have stata replace the i and sof anh sci nti meer eral network (ck Cher 9) Piceral repretatces of ll the sicmenions scree fences are hata fn Figwe LI noteworthy Hat all action hii alien bere are moots Forthermor, other than the teatity action fantom, most ofthe eter nation ‘norton atret tlre abate vals ofthe argent at whch increasing ber oer ‘notch activation ch As we wil ate, stele bettun fucns re “+ (oh ay 0) Sion (© Sim is 1 OTe (Raut (0 Mand Ta Fig 17: Varo thton fmt 2 Scanned with CamScanner scanned with CamscannerSubhahuxmi Sahoo ako very fin ater adore complex purl tw, case they belp in creating ‘wore pow companions of dierent typo of ations: Many of the faci are {ele to at squashing Fc, othe ma the pats Kem airy range to Toupdedoutpts The te of pnb ectiation plas fundamental oe i nee the modeling por ofa network. Ia network used only Hiner activations, i woud not ‘roride better modeling power than a ng yer lar neewoek. This ise is essed tection 1 1.3.2. Softmax Activation Function ‘Ts al ete ti see tha cht coups i tthe ep lager wo tap Eel val its F proto of Ueete ents or exatpe, comer the kon claiaton protien in wich ch data ror ade to be mapped to oe of ‘ued as ab sh eas outta ca be ed, with sofa eteation [oclien oh reper to rmlealecl oxtpts 8 = fe ot te mais na gia ge ‘This activation ction map ra valen to peli hat su to 1. Spee the seshaton Faction Sr the th ont be eine fll sai) Tree) ‘An example ofthe sftmax faction with thre oupats seated in Figure 18, and the sac, opal op oe len shown fn the same gm. Ne tnt the two ecg Carsespond to te putes ofthe tee clases at they covet the thee output the fal heen lage nto proubltiee with the mma feton, The fl hide ber fc wes Boar (entity) activations we iit et the sata lager. Prteraae eK = We (AD an Figure 1: An example of multiple outputs for categorical camiication withthe we of 3 stax Iyer there ae no wigs soci with the softens ae, cet sony connerting read ‘puts into probes Ea output the peoabaty of «pari chaw a Scanned with CamScanner scanned with CamscannerSubhahuxmi Sahoo 1.3.8 Common Loss Functions ‘The chief the la function rita in ening the tpt away at inset to 2 spe squared lw of Use oe rition fj. Fr proie pricion of ator dat two eypen of on te ‘read, pening on wher the piction nary ar whether al 1. Binary targets (logit regression): [ica tx an dt te seve ‘ale isda om (41) sed he peti jammed netation feet {iromtpt je (0.1), which imate th peoaity thatthe ober tale y ‘Then, the tegative arthm of |y/2 — 03 +f] rove the hw. Thien Bee [o/2 05.49, dca the probity that he precition i ort 2. Categorical targets: In this ease if jy. are the probabiitin of the Kes (cing the sft acvation of Equaon 18), ad ther clas the rout ‘eth thee feo ge te ei ello = atin) as "This ype of low fnction nphaments maltioomial logic regreion ae i ed sw the renin le Mele tok Herbie iti fo ‘mulinnl hagtc grew, we the ylue a ist to 2 the ater ‘Te ky pint to emer i that the ture of Ue outpt nade, the activation faeton, sn the knw tion depend on the appiiation at hand Figure 8: The basie architecture of a feed-forward network with two hidden layers and a single output [Link] though each unit contains a single scalar variable,one often represents all units with in a single layer as a single vector [Link] units are often represented as reetangles and have connection ma- trices between them. u Scanned with CamScanner scanned with CamscannerSubhaluxmi Sahoo Chapter 2: The Backpropagation Algorithm 1 Introduction ‘The Rosenblatts perceptron has alzeady been completed in chapter 1 that was basically a single layer neural network. Its application is limited to classifica tion of linearly separable patterns. To avoid the limitations, we have seen a new neural structure known as the multilayer perceptron, (a) The model in each neuron includes a differentiable non linear activation function. (b) The network contains one or more layers that is hidden from both the input, and the output nodes. (c) The network inciudes a very high degree of connectivity. A very popular method for training the multilayer perceptrons is the back- propagation algorithm. Thst algorithm was introduced by Rumelhart et al However several computational, stability;and overfitting challenges were found in the use of this algorithm, 1.1 The Computational graph Abstraction A computational graph is a general abstraction of a neural network and is de- fined as a directed acyclic computational graph (DACG). DACG is defined as ‘an acyclic computational graph with a directed acyelie graph of nodes where every node contains a variable. Edges might be associated with learnable pa rameters. A variable in a node is either fixed externally ( for input nodes with no incoming edges) or itis computed as a fiuetion of variables in the tail ends of edges incoming into the nodes and the learnable parameters on the incoming edges. 1.2. Computational Graphs create Complex Functions Any computational graph in a neural network evaluates composition of functions computed at individual nodes. In the above graph as seen in Fig. 1 we can Figure 1: Recursive nesting awkwardness caused by computational graph. write the expression of variable in each node in order to show the increasing complexity with the x1=F(x0,y0) Scanned with CamScanner scanned with Camscanner92) GF (F (20.90), G (30 0)),G(F0.30),G(30,90))) ‘We can easily observe here that the complexity and length of the closed form function increases exponentially with the path length in computational graph. 2 Backpropagation in Computational Graphs Backpropagation (short for backward propagation of errors) is the algorithm. used to train neural networks by updating weights based on the error between the predicted output and the actual target. It relies on the chain rule of ealeu- lus to compute gradients of the loss function with respect to the weights. Its different steps in short are: Forward pass > compute output. Compute loss. Backward pass > compute gradients via chain rule, Update weights & biases. In the single-layer neural network, the training process is relatively straight- forward becanse the error (or loss funetion) can be computed as a direct fune- tion of the weights, which allows easy gradient computation. In the ease of ‘multi-layer networks, the problem is that the loss is a complicated composition, function of the weights in earlier layers. The gradient of a composition function is computed using the backpropagation algorithm. The backprop agation algo- rithm leverages the chain rule of differential calculus, which computes the error gradients in terms of summations of local-gradient products over the various paths from a node to the output. Although this summation has an exponential number of components (paths), One can compute it efficiently using dynamic programming. ‘The backpropagation algorithm is a direct application of dy~ namie programming. It contains two main phases, referred to as the forward ‘and backward phases, respectively. The forward phase is required to compute the output values and the local derivatives at various nodes, and the backward phase is required to accumulate the products of these local values over all paths from the node to the output: 1, Forward phase: In this phase, the inputs for a training instance are fed into the neural network. This results in a forward cascade of computations across the layers, using the current set of weights. The final predicted output can be ‘compared to that of the training instance and the derivative of the loss function, with respect to the output is computed. ‘The derivative of this loss now needs Scanned with CamScanner scanned with Camscannerto be computed with respect to the weights in all layers in the backwards phase. 2. Backward phase: The main goal of the backward phase is to learn the gradi ent of the loss function with respect to the different weights by using the chain rule of differential calculus. These gradients are used to update the weights Since these gradients are learned in the backward direction, starting from the output node, this learning process is referred to as the backward phase. Hence, weights are updated in the negative direction of the gradient. This process is called as reverse engineering the loss function or the gradient of the loss function, to update the weights. One cycles through the training points repeatedly until convergence is reached. A single cycle through all training points is termed an epoch. Although we have to compute the gradient of the loss function with respect to ‘weights in a computational graph, it turns out that derivatives of node variables with respect to one another can be easily used to compute the derivative of the loss function with respect to weights on the edges. ‘This is termed the Chain rule of computing node to node derivatives dflg(z)) _ af(al=)) alate) ‘d(z)~ dta(2)) d(x) According to the above equation, for any pair of source-sink nodes, the multi- variate chain rule can be recursively applied to derive the fact that the derivative of the variable in the sink node with respect to that in the source node is simply the sum of the expressions arising from the univariate chain rule being applied to all paths existing between any pair of nodes. Specifically, it leads to the following exponential time algorithm to compute the derivative of the output o with respect to variable x in the graph, a) 2.1 Pathwise Aggregation Lemma Lemma 2.3.1 (Pathwise Aggregation Lemma) Consider a dicted acylic compte tional graph in which the 1b nade contains saree yi). The ll derivative s(,f) af the Aircel edge (i,j) im the graph ts defined 0 243) = ul set of paths P att from a node # the groph ta node t. Then, the vale af 8° given by compating the product ofthe lel gradients long wach pth iP, and suring these pradacts over at pan oni) La o-5 Tu a 2.2 An exponential Time Algorithm. ‘The pathwise aggregation lemma provides a natural exponential time algorithm that is roughly similar to the steps one would go through by expressing the Scanned with CamScanner scanned with CamscannerSubhahuxmi Sahoo Figaro 25: The auuber of paths in a computational grap increases exponentially with depth, In this cme, the in rule will aggregate the product of Teal derivatives along ‘2 paths 1. Us computational graph o compute the ay) of ach nde frward pe 2 Compute tho onl partial devas (,j) = Son cach egy inh computational eh {8 Let Po the st ofall paths from an input node with value 2 to tho output o. Ror och pth P'eP compute the product fea oa dvtve ej) ou that path 4 Ad p these wales over ll paths iP ns gouera,« compatational graph wil have an expos inten numberof paths with depth andl one mst a the product ofthe Teal derivatives overall paths. A example 15 shown in Figure 25, in which we have Ave layers, each of which Ins only two units ‘Therefore, the number of paths between the input and outputs 2° = 52. The jh hidden nit ofthe th layer denoted 3). Each den ni dete ste pvt of ts ings: inj) = MG 1,1)-h(E= 1.2). YF € {1.2} 2) In this cae, the tpt 2°, which expres ned frm and ean be diferent cally with repeet to 2. In ther words, we dot really need comptatinal grap rdor to perform the difeeatiation However, wo will we tho exponential algeria 1 cuciate ts workings. The derivatives of each Bij) with tepect to ts tro inp ae the vals ofthe conaphmeary pt, beter the paral drone ofthe mateo ‘ftw varahes the complementary arial aus) eat “am pn stn nn ht he ab of sb pot of he a dacs (wh the completa mp a tare cn) ag A 2 pas ani pet othe ap SF Tiana nawma sass) - 5 ea anata M1.) ‘Thereelt of cour, condstnt with what cue woald obtain os dfrtating 2 Ucety with repect tos However an important olervation that i equre agaregations to ompte the derivative fn this way for rately sple graph. More portant, se repetedlydieretiate the sane Junction computed in a nade ox ggregation. Fo exatple {he lilerentition of the variable (3.1) i pexfrmed 16 tes ese it appears in 16 pats bmn to 0. Scanned with CamScanner scanned with CamscannerSahoo computational functions in elosed form with respect to a particular variable ‘and then differentiating them. Specifically it leads to the above exponential time algorithm to compute the derivative of the output o with respect to variable x in the graph. 3. Backpropagation in Neural Networks ‘The backpropagation algorithm equations for @2-layor neural ebwork (input hidden + output) are mentioned below. It can be extended to deeper networks by following the same recursive pattern Forward Pass ae 7a Bale (3!) ala = WHE + HG Cost Function (MSE or LMS) 20,8) = ply al? 3) Backward Pass For m training samples Lys coe J = Fy le - 91h @) Backward Propagation 81 = BF = (a? -yoo'(#) © 9 _ s2i(qlthf paver = O00) CC) oc a @ at = (WW) PSH) @ of (2) ®) ® (20) ay (12) Let us use the above equations for a multilayer perceptron with only 1 hidden layer as given in the following diagram 3. Scanned with CamScanner Scanned with CamscannerSubhaluxmi Sahoo Input Hidden Output Layer Layer Layer 1, Forward Propagation Consider « neural network with one hidden layer a wile a 3) alt = o(2t) (a4) 2 = Wall 4 pf (as) " A) (a6) where o(:) is an activation function (e.g, sigmoid, ReLU, tanh) =o(28 2. Cost Function (MSE) For one training example: Cy.8) = bly al? az) For m training examples: Yi - Or (8) 6 Scanned with CamScanner Scanned with CamscannerSubhaluxmi Sahoo 3. Backward Propagation We compute gradients with respect to the cast function Output Layer Gradients a = BE = (a —oo'() as) pe = Hay" () a ey Hidden Layer Gradients alll = (WETS) @ o'(2) (22) ac gr awn =8 @) ac oe og oun @ 4, Parameter Update (Gradient Descent) For learning rate 7 ac yi «wll — 2 Wh WE ne 5) , oc Wil (26) 5. Summary Backpropogation with MSE tavolve «# Forward pass to compute j «© Compute error using MSE loss «Backward pass using chain rule to compute 6! ¢ Gradient descent update for weights and biases. ‘The above equations can be generalized toa multilayer perceptron with L hidden layers, Scanned with CamScanner Scanned with CamscannerSubhaluxmi Sahoo 1. Forward Propagation For an Ieayer neural network, let Pace For ench layer = 1,2 vill gl. ati(s") 2. Cost Function (MSE) For a‘ingla training exomple (v.6) = bly al? For m traning examples J= LS — 01" 3. Backward Propagation Output Layer (/ = L) ws sue oot (sth) = (4 —yoott(st) OL suicgit=n1yr aig = Cay OL _ gu aa = * Hidden Layers (/= 1 ~1,...,1) For each hidden layer: B= (PAF) 0 0 ( OE git(qll1yr awn ~* (any a 7) (28) 2) (30) (a) (32) (33) a) (35) Scanned with CamScanner Scanned with CamscannerSubhaluxmi Sahoo 4. Parameter Update For learning rate 7, update each layer (= 1,2,...,L: ac Og yy WH WH a (36) ac He yt BE a (37) 5. Summary Forward pass: compute 21, a! for ‘© Compute error using MSE. Backward pass: recursively compute 6! from output to input ‘© Update parameters WY, bl using gradient descent, 4 Vector-Centric View of Backpropagation It is the vector-centric view of backpropagation is the compact, linear algebra, ‘way of writing backprop, instead of working neuron-by-neuron. This is what ‘modern deep learning libraries actually implement because it’s fast and elegant. 4.1 Vectorized backpropagation using MSE for an L-layered neural network. 1. Forward Propagation We denote: Plas For each layer = 1,2,...,L: Walall-" 4 1 (38) alll = ol (all) (39) ‘The final output is: a 2. Cost Function (MSE) For m training examples: LS wo iy F= 5 bv - 9 9 Scanned with CamScanner Scanned with CamscannerSubhaluxmi Sahoo 3. Vectorized Backward Propagation Output Layer (/ = L) 3H = (alll —y) oo") (40) Vind = Eatealt=ayr ay Vaud = bran (2) Hidden Layers (! = L —1,...,1) af = (WEFAN oo) « wed = Ealal—Hye (4) Duos = L Sooo as 4. Parameter Update For learning rate n, update each layer |= 1,2,..-.: WE WE Vyynd (46) HM Vand an 5. Summary In the vector-eentrie formulation: «Forward pass computes (2! al) sequentially. 4 Backward pass computes 6M recursively using vectorized operations. ‘© Gradients are expressed as outer products between error vectors and ac- tivations. ‘* Parameter updates are performed using gradient descent 4.2 Vector centric Backpropagation with non-layered ar- chitecture ions written in a way id ap plies to general computation graphs (as in automatic differentiation / modern Here we will sce the veetor-centrie backprops that doesn't assume a strict layer-wise arch 10 Scanned with CamScanner Scanned with CamscannerSahoo deep learning frameworks). Tn layered architectures, neurons in each layer i are connected to all the neurons in the layer +1 and layer #1, generating a computational graph that appears fsa single path, In a nom-layered architecture, the computational graph skips connections between alternate layers. The following equations deseribe the back- propagation algorithm in a vector-centric mode for the non-layered architecture that uses the general vector-Jacobian formulation used in autodiff engines (Py= ‘Torch, TensorFlow, JAX). Instead of layers, the recursion works across arbitrary raph nodes 1. Setup Consider a computational graph with nodes representing variables {v1, 02, .-.. Un} Each node is computed as a differentiable function of its predecessors: S,(parents(v,)). ‘The final scalar output is the cost funetion: I= Fle) 2. Forward Pass Evaluate the graph in topological order: vy = fy(parents(v;)), me 3. Backward Pass (Vector-Centric Form) Define the adjoint (backpropagated gradient): ar 9° dey Initialization At the output node: Recursion For each node 2, the gradient is obtained by accumulating contributions from its childven: ‘ Se ne Dae eaten) n Scanned with CamScanner Scanned with CamscannerSubhaluxmi Sahoo 4. Parameter Gradients Ifa node v, depends on trainable parameters 0, then: av od m5 PS. Ved = 5, 3B 5. Summary Farward pan compote all node aloe oy Backend pass propagates using vetorJeesblan products ‘© This formulation does not assume a layered architecture; it applies to any directed aeyelic computation graph, 5 Some useful derivatives of activation functions Scanned with CamScanner Scanned with CamscannerSubhahuxmi Sahoo All tho updates in the previous ston rq the ke ofthe deities of acon ‘ctiation fuetiows. For this eusoa, the desvativs of the aclvtion funetins age used repeatedly n this bok, This setion provides deta of thew deri. 1. Linear and sip activations: "The delvative of the Hoearsetlvatin fet i Lat All places. The deiative of sgn(e) is O at all vale of © otber than at = 0, ‘where i is dicontimets and nondiereniae. Bese of the aero gradient and ‘onda of th activation faction, arly wed In he os ction Figure 2:10: An ample of an unconventional architectre fn which inputs ocr to lagers ‘other than the Bist hidden Laer ath (@) Meaty Sign (©) Simoia nN in /\ 3 os (@ Toot (©) Rew (0 Hed Tank Figur 2.1; The derivatives of varios activation fnetions a Scanned with CamScanner scanned with CamscannerSubhahuxmi Sahoo ce whe tue fr pret tsting tne The deat of the Ha ad ‘gations are Matted in gues 211( ad (), empty Sie ctnation The dative of sl activation i partial spl, when ‘tir expesee in oso he expt of the snc, rather than the nyt Leto the cpt of these nc with npn —— Trenay ‘The, one ca write the derativ ofthe tation allows en an ‘The hey point is that this sigmoid can be written more conveniently in terms ofthe ‘outputs: Raai-o) (23) ‘The derivative of the sigmoid is often used as «function of the output rather than the input, The derivative ofthe sigmoid activation function is Mastrated in Figure 2.13(c). 3. Tank activation: As in tho cas of tho sigmoid setivation, tho tank setivation is often used asa funetion of the output o rather than the input exp(2e) =1 expe) +t — ‘One can then ccmpute the derivative ws follows: Do __4-exp(2v) Oe ~ expe) =F ‘Ono can also write this dovivaive in terms of the output: -# (2.10) ‘The derivative ofthe tan activation is illustrated in Figure 2.11(d). 4. ReLU ond hanl tank ectications: The ReL.U takes on n partial derivative vale of 1 {for non-negative values of its argument, and 0, otherwise. The hard tanh funetion takes on a partial derivative value of 1 for values of the argument in [—1,+1] and 0, otherwise. The derivatives of the ReLU and hard tanh activations are illustrated in Figures 2.11) and (f), respectively. (24) u Scanned with CamScanner scanned with CamscannerSubhahuxmi Sahoo ‘The Special Case of Softmax Soft station ia special ease boca the futon i ot computed with rsp to one Input, but with respect to malt inputs: Therefore, one camot use exactly ae sae typ ci upae, ne with ther activation ction As dco in Equation 1 of Chapt 1, ‘the softina activtion faetion fn wwetortonsector faretion that cnet yea eed ‘rodictions ey. ito output probability .--0, ang the felling relationship: expe) Fae) [Note that if metry to use tho dain rule to the derivative ofthe ks L with respect tooth Hen 006 has to eampute each and abo each $=. This backprope- fate ofthe sft atl snp, wer we a fart inte acct “ Wel) ean 1. The softmas almost always used ln the output lye 2. The softmax alnont alway pr withthe crs-entropy lus. Heya € (0,1) be th ane hot encoded (observed) outputs for tho kmtually exclusive clases, thet the eromentop Ts dened a lowe oto.) es) Note tha if we try tows the dai uot the drei of he lus L with respect to tytn; then ope ha to compte eth Sand abo each $2. The backprope ‘tha ofthe its rests sped ten we tab 0 fact it eon 1, The sta i ah bey ns inthe tpt aye 2 Tho sta ial always pir withthe erose-eivopy lose. Hy. .-t4€ (0.1. bh the onetotcaoded alert) utp forthe knell exch ls the the crow op lm eine fll: b= -Enbato) as) a by a nt han $k pty sop thc fin ane am) 6 Scanned with CamScanner scanned with CamscannerSubhaluxmi Sahoo Chapter 3: Machine Learning with Shallow Neural Networks 1 Introduction 1, Conventional machine learning often uses optimization and gradient-descent ‘methods for learning parameterized models. Examples of such models in- clude linear regression, support vector machines, logistic regression, di- mensionality reduction, and matrix factorization. 2. Neural networks are also parameterized models that are learned with con- tinuous optimization methods. This chapter will show that a wide variety of optimization-centrie methods in machine learning can be eaptured with very simple neural network architectures containing one or two layers. 3. In fact, neural networks can be viewed as more powerful versions of these simple models, with this power being achieved by combining the ba sic models into a comprehensive neural architecture (ie., computational raph). Tt is useful to show these parallels early on, as this allows the understanding of the design of a deep network as a composition of the basic units that one often uses in machine learning. 4. Complex or deep neural architectures are often an overkill in instances where only a small amount of data are available. Neural networks are an advantage where data availability is very high ‘One way of viewing deep learning models is asa stacking of simpler models like logistic or linear regression. The coupling of a linear neuron with the sigmoid activation leads to logistic regression, which will be discussed in detail in this chapter. The coupling ofa linear unit with sigmoid activation is also used! extensively for building complex neural networks. 6. The ability to put together the basic units in a clever way is a key archi- tectural skill required by practitioners in deep learning. Nevertheless, it is also important to learn the properties of the basic models in machine learning, since they are used repeatedly in deep learning as elementary units of computation 7. The prime focus will be on two classes of model for machine learning. (a) Supervised models: The supervised models discussed here primarily correspond to linear models and their variants. ‘Those include methods like least-squares regression, support vector machines, and logistic regres- sion, Multiclass variants of these models will also be seen, (b) Unsupervised models: The unsupervised models discussed here pri- marily correspond to dimensionality reduction and matrix factorization. Scanned with CamScanner scanned with CamscannerSahoo ‘Traditional methods like principal component analysis ean also be pre- sented as simple neural network architectures. Minor variations of these ‘models can provide reductions of vastly different propertis Neural Architectures for Binary Classification Models ‘The corresponding neural architectures are minor variations of the per= ceptron model in machine learning. The main difference is in the choice of the activation function used in the final layer, and the loss function used on these outputs. 1 ofthe perceptron with both discrete and cor igure 2.2: An extended architect predictions . The corresponding neural architectures are minor variations of the per- ceptron model in machine learning. The main difference is in the choice of the activation function used in the final layer, and the loss function used, con these outputs. A single-layer network with d input nodes and a single output node is considered. The coefficients of the connections from the d input nodes to the output node are denoted by W = (wi ..wd). And the bias value is taken as 1 Scanned with CamScanner Scanned with Camscannertech piace Loasntrey codth shettloe foul Was a vent rs acline Aa Boowrinn Uses conventionak ig pee obrent- descent elke] parsornod eot'2edl Sawaal sipkancvied mmole aplomb on Pog Leaning Pho : eile ABS | nee ee opt'sa key Jo chwyne | etic. 8 CON Love complesc jee eee ley Pere Kon etl 2 Jonger v avautla buntg of cate peat lade ngewe no trorks oh pestowan | en one ip thy ca | Oo le connec Le ’hey ada ple oe sj urdls poke He portephaen wm > €nammples of aden pls prod doe threo wpa P 4 hoger boc ne greeweon. Scanned with CamScanner Scanned with Camscanner4: poasdcallyy fo classes of me lol Poy map tntinter] racine Leowndnry phat can Le ee Fr pape nerotots are supeavesed amoelth fF unsupervesed anole * , Ae ee ee a ‘ai will discuss He basen archtieclare Soe made Le anntry well poee feast Squrawe ceyserstor ¢ choschea, ye. osrespordiing noanalt aechrhectome os a onbros yastiatton of He peweplaonay mode of ancl ne. Leasnt'nsy + he enptn doPberence On: the ‘mocloh ah i peo chose ch actralkt, Lanckicnt 4 Pe Unevee of foes Pane deton, WL yall be ctesuming a ednplt Reine, vetwosk with A Bnpuknocles, + Stipe ep redex \ ; wi oy (Actesteyp) mg est bo D) Vontiab' fe scanned with Camscanner Scanned with Camscanner