Basics of Machine Learning Dr. Jyotismita ChakiLearning Algorithms: Supervised * These algorithms learn from labeled data, where each input example is paired with a correct output. The goal is to learn a mapping from inputs to outputs to predict future outcomes. * Classification: * Predicts a categorical output (e.g., spam or not spam, disease or no disease). Examples: Logistic Regression, Decision Trees, Random Forests, Support Vector Machines (SVMs), Naive Bayes, K-Nearest Neighbors (KNN). + Regression: + Predicts a continuous numerical output (e.g., house prices, temperature). Examples: Linear Regression, Polynomial Regression, Ridge Regression.Learning Algorithms: Unsupervised These algorithms work with unlabeled data, aiming to discover hidden patterns or structures within the data without prior knowledge of the output. «Clustering: Groups similar data points together into clusters. *Examples: K-Means, Hierarchical Clustering, DBSCAN. *Dimensionality Reduction: Reduces the number of features or variables in a dataset while preserving important information. *Examples: Principal Component Analysis (PCA), t-Distributed Stochastic Neighbor Embedding (t-SNE). «Association Rule Learning: Discovers relationships between variables in large datasets.Learning Algorithms: Semi-Supervised *These algorithms combine aspects of both supervised and unsupervised learning, using a small amount of labeled data and a large amount of unlabeled data for training. Pseudo-labeled Unlabeled dat Inlabeled data Dataset Prediction = @ -E- 6 ML Model Mi podel labeled data data it @ _,f OE 1is Square 2is Triangle Training ModelLearning Algorithms: Self-Supervised + A subset of unsupervised learning where the model generates its own labels from the input data, often by predicting missing or corrupted parts of the data. + Self-Supervised Learning (SSL) is a type of machine learning where a model is trained using data that does not have any labels or answers provided. * Instead of needing people to label the data, the model finds patterns and creates its own labels from the data automatically. * This allows the model to learn useful information by teaching itself from the data.Learning Algorithms: Self-Supervised + In Natural Language Processing (NLP), a model might be trained to predict the next word in a sentence: * Input: "The catsatonthe___" * Target: "mat" + Here, the label ("mat") is not given by a human; it’s part of the input data.Learning Algorithms: Reinforcement * Reinforcement learning (RL) is a machine learning technique where an "agent" learns to make a sequence of decisions by performing actions in an environment to maximize a reward. +t learns through trial and error, receiving positive or negative feedback (rewards or penalties) for its actions, and uses this feedback to improve its performance over time. * This method is used for tasks like training a robot to move, playing chess, or a car learning to navigate without explicit programming.Learning Algorithms: Reinforcement Reward Best Action . . EHS ea State Selection of ies] ee AlgorithmBuilding machine learning algorithm * Step 1: Define the Problem * Understand the problem at hand and its context. * Determine the intended result or forecast that you would like the algorithm to fulfill. + Convert the issue into a machine learning task, like clustering, regression, or classification. * Step 2: Collect and Prepare the Data + Data Collection Techniques: Gather relevant data from various sources, including databases, APIs, sensors, or web scraping. Make sure the data is accurate and pertinent to the issue. + Data Cleaning and Preprocessing: Clean the data by handling missing values, removing outliers, and transforming it into a suitable format for your chosen algorithm.Building machine learning algorithm * Step 3: Choose the Right Algorithm + Overview of Common Algorithms: Learn about different algorithms such as neural networks (for complex pattern recognition), decision trees (for decision rules), and linear regression (for continuous value prediction). * Criteria for Selecting the Algorithm: Consider factors like the type of problem, data size, interpretability requirements, and computational resources when choosing an algorithm. * Step 4: Split the Data ing Set vs. Test Set: Separate your data into two sets: a test set, which is used to assess the model's performance on unseen data, and a training set, which is used to train the model * Cross-Validation Techniques: Employ techniques like k-fold cross-validation to assess model generalization further and prevent overfittingBuilding machine learning algorithm * Step 5: Train the Model + Training Process: Feed the prepared data into your chosen algorithm, allowing it to learn patterns and relationships within the data + Tuning Hyperparameters: Change the hyperparameters, which regulate the learning process, to maximize the model's performance. + Step 6: Evaluate the Model + Model Evaluation Metrics: Assess the model's performance using appropriate metrics like accuracy, precision, recall, F1 score (for classification), or mean squared error and R-squared (for regression). * Step 7: Optimize the Model + Model Optimization Techniques: Use techniques such as feature engineering, hyperparameter tuning, or ensemble methods (combining multiple models) to further enhance the model’s performance.Building machine learning algorithm + Step 8: Deploy the Model + Model Deployment Options: Deploy your model into a production environment, either on-premises or in the cloud, to make predictions or decisions on new data. + Monitoring and Maintenance: To guarantee the model's accuracy and applicability, keep a close eye on its performance and retrain it with fresh data as necessary. + Step 9: Document and Review + Importance of Documentation: Document your code, data preprocessing steps, model architecture, and evaluation results to ensure reproducibility and facilitate collaboration. * Code Reviews and Collaboration: To find areas for improvement and guarantee code quality, conduct code reviews and solicit input from mentors or peers.Biological Neuron * The foundational unit of the human brain is the neuron. + A tiny piece of the brain, about the size of grain of rice, contains over 10,000 neurons, each of which forms an average of 6,000 connections with other neurons. * It's this massive biological network that enables us to experience the world around us. * Our goal in this section will be to use this natural structure to build deep learning models that solve problems in an analogous way. * The neuron is optimized to receive information from other neurons, process this information in a unique way, and send its result to other cells. + This process is summarized in Figure 1-6.Biological Neuron Ay Tz ) a em Inputs Strengths Sum | Transform Output Figure 1-6. A functional description of a biological neuron’ structureBiological Neuron * The neuron receives its inputs along antennae-like structures called dendrites. * Each of these incoming connections is dynamically strengthened or weakened based on how often it is used (this is how we learn new concepts!), and it’s the strength of each connection that determines the contribution of the input to the neuron’s output. + After being weighted by the strength of their respective connections, the inputs are summed together in the cell body. * This sum is then transformed into a new signal that’s propagated along the cell's axon and sent off to other neurons.The Neuron + We can translate this functional understanding of the neurons in our brain into an artificial model that we can represent on our computer. Such a model is described in Figure 1-7. + Just as in biological neurons, our artificial neuron takes in some number of inputs, x = [x1X2,"**%n), each of which is multiplied by a specific weight, w = [wv,,W2,-+, Wn]. + These weighted inputs are, as before, summed together to produce the fogit of the neuron, z = Wixi. *+ In many cases, the logit also includes a bias, which is a constant (not shown in the figure). + Bias helps in controlling the value at which activation function will trigger. *+ The logit is then passed through a function f to produce the output y = f(z) or f(x-w +b), where b is the bias term. + This output can be transmitted to other neurons. + In other words, we can compute the output by performing the dot product of the input and weight vectors, adding in the bias term to produce the logit, and then applying the transformation function.The Neuron x, Ww, x, Wy | : xX, ————_-w, Figure 1-7. Schematic for a neuron in an artificial neural netNeural Network * Neural networks are machine learning models that mimic the complex functions of the human brain. * These models consist of interconnected nodes or neurons that process data, learn patterns and enable tasks such as pattern recognition and decision-making. * Neural networks are capable of learning and identifying patterns directly from data without pre-defined rules.Neural Network * These networks are built from several key components: + Neurons: The basic units that receive inputs, each neuron is governed by a threshold and an activation function. * Connections: Links between neurons that carry information, regulated by weights and biases. + Weights and Biases: These parameters determine the strength and influence of connections. * Propagation Functions: Mechanisms that help process and transfer data across layers of neurons. + Learning Rule: The method that adjusts weights and biases over time to improve accuracy.Hidden Lay Neural Network See Input Layer e) «) Output Layer [SO SSIS (QOL SKLKF OSS EE RIERA SSE PERRIER SRNeural Network + Input Layer: This is where the network receives its input data. Each input neuron in the layer corresponds to a feature in the input data. * Hidden Layers: These layers perform most of the computational heavy lifting. A neural network can have one or multiple hidden layers. Each layer consists of units (neurons) that transform the inputs into something that the output layer can use. * Output Layer: The final layer produces the output of the model. The format of these outputs varies depending on the specific task like classification, regression.Linear separability * The concept of separability applies to binary classification problems. In them, we have two classes: one positive and the other negative. We say they’re separable if there’s a classifier whose decision boundary separates the positive objects from the negative ones. + If such a decision boundary is a linear function of the features, we say that the classes are linearly separable. * Since we deal with labeled data, the objects in a dataset will be linearly separable if the classes in the feature space are too. + We say a two-dimensional dataset is linearly separable if we can separate the positive from the negative objects with a straight line.Alinear boundary @ Positive objects. Negative objects Linear separability a ° n Ss; wyxj,t+b x i=lExpressing Linear Perceptron as Neurons * Consider the neuron depicted in Figure 1-8. * The neuron has two inputs, a bias, and uses the function: -1 ifz0 fR=1 4 ifzs0 + It’s very easy to show that our linear perceptron and the neuronal model are perfectly equivalent. * And in general, it’s quite simple to show that singular neurons are strictly more expressive than linear perceptron. «In other words, every linear perceptron can be expressed as a single neuron.Expressing Linear Perceptron as Neurons xi My, sleep study Figure 1-8. Expressing our exam performance perceptron as a tteuronGradient Descent Cost Minimum Random initial value @>Gradient Descent Initial Weight. ‘Gf Gradient Incremental ce Minimum Cost Derivative of Cost WeightStochastic Gradient Descent * Stochastic Gradient Descent (SGD) is a widely used optimization algorithm for training neural networks, which updates the model's weights using a single data point or a small mini-batch at a time, rather than the entire dataset. * This approach makes the training process much faster and more computationally efficient, especially for large datasets. * While the updates are noisier than with traditional batch gradient descent, this randomness can help the model escape local minima and find a better overall solution.Stochastic Gradient Descent [Link]: The neural network starts with a random set of parameters (weights). [Link] a sample: A single data point is randomly selected from the training set. [Link] the gradient: The loss function is computed, and the gradient is calculated with respect to the loss for that single sample or mini-batch. [Link] weights: The network's weights are updated in the direction that minimizes the loss, based on the calculated gradient. [Link]: This process is repeated for many more samples or mini- batches until the model has converged.Stochastic Gradient Descent Global Minimum Local MinimumMultilayer Perceptron +A multilayer perceptron is a type of feedforward neural network consisting of fully connected neurons with a nonlinear kind of activation function. + It is widely used to distinguish data that is not linearly separable. *MLPs have been widely used in various fields, including image recognition, natural language processing, and speech recognition, among others. * Their flexibility in architecture and ability to approximate any function under certain conditions make them a fundamental building block in deep learning and neural network research.Multilayer Perceptron + Input layer * The input layer consists of nodes or neurons that receive the initial input data. Each neuron represents a feature or dimension of the input data. The number of neurons in the input layer is determined by the dimensionality of the input data. * Hidden layer * Between the input and output layers, there can be one or more layers of neurons. Each neuron in a hidden layer receives inputs from all neurons in the previous layer (either the input layer or another hidden layer) and produces an output that is passed to the next layer. The number of hidden layers and the number of neurons in each hidden layer are hyperparameters that need to be determined during the model design phase. * Output layer * This layer consists of neurons that produce the final output of the network. The number of neurons in the output layer depends on the nature of the task. In binary classification, there may be either one or two neurons depending on the activation function and representing the probability of belonging to one class; while in multi-class classification tasks, there can be multiple neurons in the output layer.Multilayer Perceptron + Weights + Neurons in adjacent layers are fully connected to each other. Each connection has an associated weight, which determines the strength of the connection. These weights are learned during the training process. * Bias neurons * In addition to the input and hidden neurons, each layer (except the input layer) usually includes a bias neuron that provides a constant input to the neurons in the next layer. Bias neurons have their own weight associated with each connection, which is also learned during training. + The bias neuron effectively shifts the activation function of the neurons in the subsequent layer, allowing the network to learn an offset or bias in the decision boundary, By adjusting the weights Connected to the bias neuron, the MLP can learn to control the threshold for activation and better fit the training data. * Activation function * Typically, each neuron in the hidden layers and the output layer applies an activation function to its weighted sum of inputs. Common activation functions include sigmoid, tanh, ReLU (Rectified Linear Unit), and softmax. These functions introduce nonlinearity into the network, allowing it to learn complex patterns in the data,Multilayer PerceptronBackpropagation @ Hidden layer(s) ; = Output layer Differencefn lesired values + "eas, Backprop output layerBackpropagation [Link] X, arrive through the preconnected path [Link] is modeled using real weights W. The weights are usually randomly selected. [Link] the output for every neuron from the input layer, to the hidden layers, to the output layer. [Link] the error in the outputs: Error,= Actual Output — Desired Output [Link] back from the output layer to the hidden layer to adjust the weights such that the error is decreased.Backpropagation: Example Target Value T1=0.01 T2=0.99Backpropagation: Example * Now, we first calculate the values of H1 and H2 by a forward pass. * To find the value of H1 we first multiply the input value from the weights as . H1=x1xw,+x2xw,+b1 H1=0.05x0.15+0.10x0.20+0.35 H1=0.3775 * To calculate the final result of H1, we performed the sigmoid function as 1 Ha =— HAgnat =——z lt gars final = 0.593269992Backpropagation: Example * We will calculate the value of H2 in the same way as H1 . H2=x1xw,+x2xw,t+b1 H2=0.05x0.25+0.10x0.30+0.35 H2=0.3925 * To calculate the final result of H1, we performed the sigmoid function as H2gnal = —- 1+ oar 1 1 1+ Soaees H2sinal = H2 final = 0.596884378Backpropagation: Example * Now, we calculate the values of y1 and y2 in the same way as we calculate the H1 and H2 + To find the value of y1, we first multiply the input value i.e., the outcome of H1 and H2 from the weights as . yl=H1xw,+H2xwetb2 y1=0.593269992x0.40+0.596884378x0.45+0.60 y1=1.10590597 + To calculate the final result of y1 we performed the sigmoid function as : te iqaat = 0.75136507Backpropagation: Example * We will calculate the value of y2 in the same way as y1 . y2=H1xw,+H2xw,t+b2 y2=0.593269992x0.50+0.596884378x0.55+0.60 y2=1.2249214 * To calculate the final result of H1, we performed the sigmoid function as Y2final sae [oe l+on 1 ¥2tinal = T 1+ ome Y2sinal = 0.772928465Backpropagation: Example * Our target values are 0.01 and 0.99. Our y1 and y2 value is not matched with our target values T1 and T2. + Now, we will find the total error, which is simply the difference between the outputs from the target outputs. The total error is calculated as Etotal = E> (target — output)?Backpropagation: Example * So, the total error is A, a, i i. = 7 (tt — lana) +5 (T2 — y2tinat)® 1 1 : = 5 (0.01 — 0.75136507)* +5 (0.99 ~ 0.72928465)* = 0.274811084 + 0.0235600257 Eotal = 029837111Backpropagation: Example + Now, we will backpropagate this error to update the weights using a backward pass. * To update the weight, we calculate the error correspond to each weight with the help of a total error. The error on weight w is calculated by differentiating total error with respect to w. OE total Errory = hid dwBackpropagation: Example * We perform backward process so first consider the last weight w5 as 1 = of f Exotal = 5 (TL —Ylaimal)? +5 (12 — y2inat)? oe wees (2) * From equation two, it is clear that we cannot partially differentiate it with respect to w5 because there is no any wS. We split equation one into multiple terms so that we can easily differentiate it with respect to w5 as AE wead _ FEvoea) _ By1final yt x x dw5 yin Oyl=S OWSBackpropagation: Example + Now, we calculate each term one by one to differentiate E,,.., with respect to w5 as Yinwal = 75 gsi" (8) 1 a Arora _ 9G (T1—Y1ival)? +3 (12 — y2snal)?) bvima Cee) @y1pnal @y1pnat ayl yt 1 2-2 cae = 2X5% (Tyna)? +X (1) +0 "aro = H{T1 yl) Se lame 1 = —(0.01 — 0.75136507) Vitel = TPF OE coral Seen, = 0.74136507 vase (4) OT finat 1 AE Ysa ene (7)Backpropagation: Example + Putting the value of eY! in equation (6) 1 fina! = = ~ io Yim final (yi 3)? V1 = Ht gina X WS + H2 pinay XW6 FB one ee ee eee (9) Yttinal Ay1 _ A(H1fnat X WS + H2final X w6 + b2) = ylana X (1 — y1einal) cue ows = Bl gna = 0.75136507 x (1 — 0.75136507) coe cat = 0.596884378 ... Oy1 final ows yl ont = 0.186815602...Backpropagation: Example SE total Ovifinal 4 Bvt : So, we put the values of 57,21’ ay1 * 24 aws_ in equation no (3) to find the final result. FErorat — OErotal , Pena, OV1 awe & ihe Styl aws = 0.74136507 X 0.186815602 x 0.593269992 OE rotal dws = 0.0821670407..........(11) Errotys =Backpropagation: Example Now, we will calculate the updated weight w5,ey with the help of the following formula PEroral ows = 0.4—0.5 x 0.0821670407 WSnew = WS — X Here, n = learning rate = 0.5 WS now = 0.35891648......... (12)Backpropagation: Example * In the same way, we calculate W6,.W7 new ANd W8pey, and this will give us the following values. . W5 ,ew=0.35891648 new! W6 ,ew=408666186 new’ W7 new2=0-511301270 W8 ,.=0.561370121 new’Backpropagation: Example + Now, we will backpropagate to our hidden layer and update the weight w1, w2, w3, and w4 as we have done with w5, w6, w7, and w8 weights. * We will calculate the error at w1 as PE vocal awl Error; = 1 4 i Erotal = 3 (T1— yl fina)” + 5 (T2 — y2tinat)”Backpropagation: Example + We cannot partially differentiate it with respect to w1 because there is no any w1. sata _ Bort OHifinal, 2H yg) Qwi OH Afinat Ht dwt + Now, we calculate each term one by one to differentiate E,,,., with respect to w1 as 1 Bae roar _ OCG (T1—Yl sina)? + 3 (T2 — y2sinel)2) FAlfnal Hi - CaBackpropagation: Example * We again split this because there is no any H1*"@! term in Et! as FErotad OE aE, me GH1pnat OHI finai PH 1 final will again split because in E1 and E2 there is no H1 term. Splitting is done as 2B, OE; 2H final 8H1final oe, _ 961 _dyt OH1gnal OY1 OH1gnat OE _ OE2, aya OH1finat OY2 OH1_final “~~Backpropagation: Example We again Split both and on because there is no any y1 and 'y2 term in E1 and E2. We split it as 2B, _ 2 Aya yt Ayla A¥T dE, dE, Oy2 OY 2siantBackpropagation: Example Now, we find the value of a and S nn by putting values in equation (18) and (19) as From equation (18) OE, OE, Oy lina dyl Oyleani Yl i : — 2G C1 vlan") | YL pnat Oyama oyl Oy 1 Gnal = 2 5(T1~Ygnat) X (-) xBBackpropagation: Example From equation (8) 1 = 2.x 5(0.01 — 0.75136507) x (—1) x 0.186815602 oF _ 9 138498562 (20) ayt 7° aoBackpropagation: Example * From equation (19) dE, _ _ OE, Oy2Gnad Gy2 By2pa YZ ai _ 9G (T2 = y2sinat)) 4 ByPtinat Oy 2sinal ay2 1 =2x—(T2 — y2, a XS (T2— vig) x1) x SYze ne(at) Y2final = moos sogg 2 MD y ay2rant OC Fe?) ay2 ay2 oe “ate =e? X (y2Final)® 4 ¥tnal = Tp ove wv (24)Backpropagation: Example + Putting the value of e in equation (23) 1 = y2sinal ; =—— xk @f - adil (y2einai) = y2tinal X (1 — y2finai) = 0.772928465 x (1 — 0.772928465) Oy2 rian = 0.175510053..........(25 ay2 (25)Backpropagation: Example * From equation (21) 1 = 2.x 5 (0.99 ~ 0.772928465) x (—1) x 0.175510053 it = —0.0380982366126414.........(26)Backpropagation: Example * Now from equation (16) and (17) ot ae, _ a; aya 9y1* tina ieee (pn X We + H2pnalX We + 2) a 2€F1t gna X Ws + Hal X We + B2) CHa X wy +H +2 ASPAPOREA OHI pna = -0.0380982366126414 x 26 a 7 Tia (HA pnat X Ws + Hpnat We + 62) = 0.138498562 x —0.0380982366126414 x w? OH Tenal 138498562 x w5 0.0380982366126414 X 0.50 = 0,138498562 x 0.40 —0,0190491183063207......(28) aE, OH final 0553994248 .......(27)Backpropagation: Example @E Ey 4 : Put the value of Minna equation (15) as —t ng—* @H1 ginal @H1 pn: PE rorat _ Ey OE» OH1gnat FHI final PHL Gna = 0.0553994248 + (—0.0190491183063207) PE coral = 0.0364908241736793 ... . OHA gina (29)Backpropagation: Example OE rota o We have #81. we need to figure [Link] | 9H a. Hi ginal Hl’ Swi 1 BHtgna °GR ED OH1 OHI am “GFemp eo X (1 gna)? » (30) Hl ginal = 1 Tfemt G1)Backpropagation: Example + Putting the value of e" in equation (30) 1 — Hl final 2 = ———— x (H1 Te (H1ginai) = Alpina X (1 — H1ginai) = 0.593269992 x (1 — 0.593269992) OH1 ginal oH1 = 0.2413007085923199Backpropagation: Example + We calculate the partial derivative of the total net input to H1 with respect to w1 the same as we did for the output neuron: H1 = H ping X WS + H2 pinay X W6 $2 ooo sse ses ws (32) yt _ A(x1 x wi + x2 xX w3 + b1 x 1) owl dw1 =x1Backpropagation: Example OHi final aoe éH1 ’ So, we put the values of 2 and = in equation (13) to find the final result. w @Erorat _— OEtora_ | AH Tnar | OHI Awl OH1gpat OHI «OWL = 0.0364908241736793 Xx 0.2413007085923199 x 0.05 ool Errotye; = = 0.000438568 ........(34)Backpropagation: Example + Now, we will calculate the updated weight w1.,,.,, with the help of the following formula Ey. otal co j “ 4 Here y = learning rate = 0.5 Wlrew = W1 —1 X = 0.15 — 0.5 x 0.000438568 Wlyew = 0.149780716......... (35)Backpropagation: Example + In the same way, we calculate W2,,.y,W3,ew and w4 and this will give us the following values . W1,,¢y=0-149780716 W2,,ew=0-19956143, w3, w4, * We have updated all the weights. We found the error 0.298371109 on the network when we fed forward the 0.05 and 0.1 inputs. In the first round of Backpropagation, the total error is down to 0.291027924. After repeating this process 10,000, the total error is down to 0.0000351085. At this point, the outputs neurons generate 0.159121960 and 0.984065734 i.e., nearby our target value when we feed forward the 0.05 and 0.1.Backpropagation: Weight updating using linear function 2 OLS yy ow bez “ila yy ee a eRe wz ey) OE 1S = dy eye wa wrE ee)Backpropagation: Weight updating using sigmoid activation function OE _@ 1 2 a. 2a Awi=— SE = 7 (~osdoult~ 01) ai OWI aD ie wan (eon) or— on) 2 Ai= le oon od Wi wrt (te 08) onl 01) x Continue. ==) (w 00) 041-0) xBackpropagation: Example2 * Consider muttitayer feed-forward neural network. Let the learning rate be 0.9 and the activation function Sigmoid activation function, The initial weight and bias values of the network are given in Table 6.3, along with the first training tuple, X= (1, 0, 1), whose class label is 1. Wis a5 Wg Mas Way W35 Wag Wag &% OO 0.2 -03. 0.40.1 2. 3 -0.2 -04 0.2 0.1 Table 6.3 Initial input, weight, and bias values. * Using back propagation algorithm, calculate the updated weight and bias value. * Check the error using the new weight and bias value.Backpropagation: Example2Backpropagation: Example2 Unit Bias weight value ; Net input, / Output, O; 0.2+0-0.5-04 =-0.7 1/0. + £7) = 0.332 -0.3+04+0.24+0.2=0.1 1 + 1) = 0.525 (-0.3)(0.332) — (0.2)(0.525) + 0.1 =-0.105 | 1/1 + €%:195) = 0.474 Table 6.4 The net input and output calculations.Backpropagation: Example2 Error Sq (ta — 0a) a= Oa) ong pen} (0.474) (1 — 0.474) — 0.474) = 0.1311 (0.525)(1 — 0.525)(0.1311)(—0.2) = —0.0065 (0.332)(1 — 0.332)(0.1311)(—-0.3) = —0.0087 04 Go) G = 04) X i Table 6.5 Calculation of the error ateach node. Backpropagated weightBackpropagation: Example2 Backpropagated Sackpropagated Old weight LR error output 7 0.3 + (0.9)(0.13 1110. 0.2 + (0.9)(0.13 11,0: 0.2 + (0.9)(-0.0087)(1) = 0.192 0.3-+ (0.9)(-0.0065)(1)= 0.306 0.4 +(0.9(-0.0087)(0) = 0.4 0.1 + (0.9(-0.0065)(0) = 0.1 0.5 + (0.9)(-0.0087)(1) = -0.508 0.2 + (0.9)(-0.0065)(1) = 0.194 0.1 + (0.9)(0.1311) = 0.218 0.2 + (0.9}(-0.0065) = 0.194 0.4 + (0.9)(-0.0087) = -0.408 Old bias -——] weight value Table 6.6 Calculation for weight and bias updating.Common Activation Functions and Their Derivatives Activation Function Formula Sigmoid a(x) LL Tanh RelU ReLU(c) Leaky ReLU LeakyReLU(z) { Derivative o'(z) = a(z)(1 ~ o(z)) tanh'(z) = 1 — tanh?(x) ReLU'(x) ' . cz LeakyReLU'(z) {Curse of Dimensionality * Curse of Dimensionality refers to the phenomenon where the efficiency and effectiveness of algorithms deteriorate as the dimensionality of the data increases exponentially. + In high-dimensional spaces, data points become sparse, making it challenging to discern meaningful patterns or relationships due to the vast amount of data required to adequately sample the space. * Curse of Dimensionality significantly impacts machine learning algorithms in various ways. It leads to increased computational complexity, longer training times, and higher resource requirements. Moreover, it escalates the risk of overfitting and spurious correlations, hindering the algorithms’ ability to generalize well to unseen data.Curse of Dimensionality *To overcome the curse of dimensionality, you can consider the following strategies: + 1. Dimensionality Reduction Techniques: + Feature Selection: Identify and select the most relevant features from the original dataset while discarding irrelevant or redundant ones. This reduces the dimensionality of the data, simplifying the model and improving its efficiency. + Feature Extraction: Transform the original high-dimensional data into a lower-dimensional space by creating new features that capture the essential information. Techniques such as Principal Component Analysis (PCA) and t- distributed Stochastic Neighbor Embedding (t-SNE) are commonly used for feature extraction.Curse of Dimensionality *To overcome the curse of dimensionality, you can consider the following strategies: + 2. Data Preprocessing: + Normalization: Scale the features to a similar range to prevent certain features from dominating others, especially in distance-based algorithms. + Handling Missing Values: Address missing data appropriately through imputation or deletion to ensure robustness in the model training process. + 3. Regularization: Utilize techniques such as L1 or L2 regularization in models like linear regression to penalize complex models and mitigate overfitting.