Scribd
Upload
5 views52 pages

Module 4

Bayesian Learning is a method that utilizes Bayes' theorem to represent and reason about knowledge in uncertain domains, making it effective for applications like game theory and medicine. It involves concepts such as prior, likelihood, and posterior probabilities, and includes algorithms like Naive Bayes for classification tasks. The document also discusses the process of calculating probabilities and making predictions based on observed data using Bayesian inference.

Uploaded by

myselfdeepikaaa2
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF or read online on Scribd
5 views52 pages

Module 4

Bayesian Learning is a method that utilizes Bayes' theorem to represent and reason about knowledge in uncertain domains, making it effective for applications like game theory and medicine. It involves concepts such as prior, likelihood, and posterior probabilities, and includes algorithms like Naive Bayes for classification tasks. The document also discusses the process of calculating probabilities and making predictions based on observed data using Bayesian inference.

Uploaded by

myselfdeepikaaa2
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF or read online on Scribd
Cl Tele) rua} Bayesian Learning “In science, progress is possible. In fact, if one believes in Bayes’ theorem, scientific progress is inevitable as predictions are made ‘and as beliefs are tested and refined. — Nate Silver Bayesian Learning is a learning method that describes and represents knowledge in an uncertain, domain and provides a way to reason about this knowledge using probability measure. It uses Bayes theorem to infer the unknown parameters of a model. Bayesian inference is useful in many applications which involve reasoning and diagnosis such as game theory, medicine, ete. Bayesian inference is much more powerful in handling missing data and for estimating any uncertainty in predictions. Reva Ae) ives ‘+ Understand the basics of probability-based learning and probability theory + Lear the fundamentals of Bayes theorem *' Introduce Bayes Classification models such as Brute Force Bayes learning algorithm, Bayes Optimal classifier, and Gibbs algorithm * Introduce Naive Bayes Classification models that work on the principle of Bayes theorem + Explore the Naive Bayes classification algorithm * Study about Naive Bayes Algorithm for continuous attributes using Gaussian distribution ‘+ Introduce other popular types of Naive Bayes classifiers such as Bernoulli Naive Bayes classifier, Multinomial Naive Bayes classifier, and Multi-class Naive Bayes classifier 8.1 INTRODUCTION TO PROBABILITY-BASED LEARNING Probability-based learning is one of the most important practical learning methods which combines prior knowledge or prior probabilities with observed data. Probabilistic learning uses the concept of probability theory that describes how to model randomness, uncertainty, and noise to predict future events. It is a tool for modelling large datasets and uses Bayes rule to infer unknown quantities, predict and learn from data. Ina probabilistic model, randomness plays a major role which gives probability distribution a solution, while in a deterministic model there is no randomness and —_— Bayesian Learning + 235 hence it exhibits the same initial conditions every time the model is run and is likely to get a single possible outcome as the solution. Bayesian learning differs from probabilistic learning as it uses subjective probabilities (ie, probability that is based on an individual's belief or interpretation about the outcome of an event and it can change over time) to infer ‘parameters of a model. Two | practical learning algorithms called Naive Bayes learning and Bayesian Belief Network (BBN) form the major part of Bayesian learning. These algorithms use prior probabilities and apply Bayes rule to infer useful information. Bayesian Belief Networks (BBN) is explained in detail in Chapter 9. ‘Scan for information on “Probability Theory’ and for ‘Additional Examples’ 8.2 FUNDAMENTALS OF BAYES THEOREM Naive Bayes Model relies on Bayes theorem that works on the principle of three kinds of probabil- ities called prior probability, likelihood probability, and posterior probability. Prior Probability Itis the general probability of an uncertain event before an observation is seen or some evidence is collected. It is the initial probability that is believed before any new information is collected. Likelihood Probability Likelihood probability is the relative probability of the observation occurring for each class or the sampling density for the evidence given the hypothesis. It is stated as P (Evidence | Hypothesis), which denotes the likeliness of the occurrence of the evidence given the parameters. Posterior Probability It is the updated or revised probability of an event taking into account the observations from the training data. P (Hypothesis | Evidence) is the posterior distribution representing the belief about the hypothesis, given the evidence from the training data. Therefore, Posterior probability ~ prior probability + new evidence 8.3 CLASSIFICATION USING BAYES MODEL Naive Bayes Classification models work on the principle of Bayes theorem. Bayes’ rule is a mathe- matical formula used to determine the posterior probability, given prior probabilities of events. Generally, Bayes theorem is used to select the most probable hypothesis from data, considering both prior knowledge and posterior distributions. It is based on the calculation of the posterior probability and is stated as: P (Hypothesis h | Evidence E) where, Hypothesis h is the target class to be classified and Evidence E is the given test instance. 236 + Machine Learning P (Hypothesis h! Evidence £).is calculated from the prior probability P (Hypothesis h), the likelihood probability P (Evidence E | Hypothesis h) and the marginal probability P (Evidence F), Itcan be written as: P(Evidence E) where, P (Hypothesis h) is the prior probability of the hypothesis h without observing the training data or considering any evidence. It denotes the prior belief or the initial probability that the hypothesis h is correct. P (Evidence E) is the prior probability of the evidence E from the training dataset without any knowledge of which hypothesis holds. It is also called the marginal proba- bility. P (Evidence E | Hypothesis h) is the prior probability of Evidence E given Hypothesis h, It is the likelihood probability of the Evidence E after observing the training data that the hypothesis / is correct. P (Hypothesis h | Evidence £) is the posterior probability of Hypothesis h given Evidence E. Itis the probability of the hypothesis / after observing the training data that the evidence E is correct. In other words, by the equation of Bayes Eq. (8:1), one can observe that: Posterior Probability « Prior Probability x Likelihood Probability Bayes theorem helps in calculating the posterior probability for a number of hypotheses, from which the hypothesis with the highest probability can be selected. ‘This selection of the most probable hypothesis from a set of hypotheses is formally defined as Maximum A Posteriori (MAP) Hypothesis. P (Hypothesis h | Evidence E) = (81) Maximum A Posteriori (MAP) Hypothesis, Rysp Givena set of candidate hypotheses, the hypothesis which has the maximum value is considerea the maximum probable hypothesis or most probable hypothesis. This most probable hypothesis is calle the Maximum A Posteriori Hypothesis h,,,.- Bayes theorem Eq. (8.1) can be used to find the yyy. Ip = MAX,., P(Hypothesish| Evidence E) ax._, P(Evidence E |Hypothesish)P(Hypothesis i) ‘at (Evidence E) =max,,, P(Evidence E | Hypothesis h)P( Hypothesis h) 62) Maximum Likelihood (ML) Hypothesis, hy, Given a set of candidate hypotheses, if every hypothesis is equally probable, only P(E | it) is used to find the most probable hypothesis. The hypothesis that gives the maximum likelihood for P (E | H) is called the Maximum Likelihood (ML) Hypothesis, hy. Ig =™AX,,, P(Evidence E | Hypothesis h) 63) Correctness of Bayes Theorem Consider two events A and B in a sample space S. ATFITFTTF BFTTFTFTF P(A)=58 PB)=4/8 236» Machine Learning P (Hypothesis /s| Evidence E).is calculated from the prior probability P (Hypothesis i), the likelihood probability P (Evidence E |Hypothesis h) and the marginal probability P (Evidence £), It can be written as: P (Hypothesis | Evidence £) = P¢Evidence ElHypothesis h) P(Hypothesis 1) (Evidence E) where, P (Hypothesis h) is the prior probability of the hypothesis h without observing the training data or considering any evidence. It denotes the prior belief or the initial probability that the hypothesis h is correct. P (Evidence E) is the prior probability of the evidence E from the training dataset without any knowledge of which hypothesis holds. It is also called the marginal probs. bility. P (Evidence E | Hypothesis ht) is the prior probability of Evidence E given Hypothesis 1, It is the likelihood probability of the Evidence E after observing the training data that the hypothesis h is correct. P (Hypothesis ht | Evidence E) is the posterior probability of Hypothesis given Evidence E. Itis the probability of the hypothesis h after observing the training data that the evidence F is correct. In other words, by the equation of Bayes Eq. (8.1), one can observe that: Posterior Probability a Prior Probability x Likelihood Probability Bayes theorem helps in calculating the posterior probability for a number of hypotheses, from which the hypothesis with the highest probability can be selected. This selection of the most probable hypothesis from a set of hypotheses is formally defined as ‘Maximum A Posteriori (MAP) Hypothesis. a) Maximum A Posteriori (MAP) Hypothesis, hy, Given a set of candidate hypotheses, the hypothesis which has the maximum value is considered as the maximum probable hypothesis or most probable hypothesis. This most probable hypothesis is called the Maximum A Posteriori Hypothesis li,» Bayes theorem Eq, (8.1) can be used to find the ly. Tiyyp = MAX, P(Hypothesishl Evidence E) P(Evidence E \Hypothesis h)P(Hypothesis h) ‘P( Evidence E) = max, P(Evidence E | Hypothesis h)P(Hypothesis h) 62) Maximum Likelihood (ML) Hypothesis, hy, Given a set of candidate hypotheses, if every hypothesis is equally probable, only P (E | it) is used to find the most probable hypothesis. The hypothesis that gives the maximum likelihood for P(E |) is called the Maximum Likelihood (ML) Hypothesis, hy, Ing = Max,,, P(Evidence E | Hypothesis h) 63) = max, Correctness of Bayes Theorem Consider two events A and B in a sample space S. ATFTTETTF BFTTFTFTF P(A)=5/8 P(@)=4/8 ——_—. Bayesian Learning + 237 P(AIB)=2/4 P(BIA)=2/5 P(A|B)=P(BIA)P(A)/P(B)== 2/4 P(BIA)=P(A 1B) P(B)/P(A)== 2/5 Let us consider a numerical example to illustrate the use of Bayes theorem now: ——— Consider a boy who has a volleyball tournament on the next day, but today he feels, ‘sick. It is unusual that there is only a 40% chance he would fall sick since he is a healthy boy. Now, Find the probability of the boy participating in the tournament. The boy is very much interested in volley ball, so there is a 90% probability that he would participate in tournaments and 20% that he \will fall sick given that he participates in the tournament. Solution: P (Boy participating in the tournament) = 90% P (Heis sick | Boy participating in the tournament) = 20% P (He is: 40% The probability of the boy participating in the tournament given that he is sick is: P (Boy participating in the tournament | He is sick) = P (Boy participating in the tournament) xP (He is sick | Boy participating in the tournament)/P (He is Sick) P (Boy participating in the tournament | He is sick) = (0.9 x 0.2)/0.4 45 Hence, 45% is the probability that the boy will participate in the tournament given that he is sick. = (One related concept of Bayes theorem is the principle of Minimum Description Length (MDL). The minimum description length (MDL) principle is yet another powerful method like Occam's razor principle to perform inductive inference. It states that the best and most probable hypothesis is chosen for a set of observed data or the one with the minimum description. Recall from Eq. (8.2) Maximum A Posteriori (MAP) Hypothesis, Ii,» which says that given a set of candidate hypotheses, the hypothesis which has the maximum value is considered as the maximum probable hypothesis or most probable hypothesis. Naive Bayes algorithm uses the Bayes theorem and applies this MDL principle to find the best hypothesis for a given problem. Let us clearly understand how this algorithm works in the following Section 8.3.1. 8.3.1 NAIVE BAYES ALGORITHM Iisa supervised binary class or multi class classification algorithm that works on the principle of Bayes theorem. There is a family of Naive Bayes classifiers based on a common principle. These algorithms classify for datasets whose features are independent and each feature is assumed to be given equal weightage. It particularly works for a large dataset and is very fast. It is one of the most effective and simple classification algorithms. This algorithm considers all features to be independent of each other even though they are individually dependent on the classified object. Each of the features contributes a probability value independently during classification and hence this algorithm is called as Naive algorithm. Some important applications of these algorithms are text classification, recommendation system and face recognition. 238. + Machine Learning ‘Compute the prior probability for the target class. ‘Compute Frequency matrix and likelihood Probability for each of the feature. Use Bayes theorem Eq. (8.1) to calculate the probability of all hypotheses. ‘Use Maximum A Posteriori (MAP) Hypothesis, li,» Eq- (8.2) to classify the test object to the hypothesis with the highest probability. sere oe Assess a students performance using Naive Bayes algorithm with the dataset provided in Table 8.1. Predict whether a student gets a job offer or not in his final year of the course, Table 8.1: Training Dataset or mr Deere ee ete ca 1 | 2 [Yes Very good Good Yes 2 | 28 [No Good Moderate Yes 3 | » [No “Average Poor No 4 8 [No Average Good No 5 | 38 [yes Good Moderate Yes 6 [| @ [ye Good Moderate Yes 7 | @ [Yes Good Poor No & | 2 [No Very good Good Yes 3 | 28 [Yee Good Good Ye | qo. | 28 [Yes “Average Good Yes Solution: The training dataset T consists of 10 data instances with attributes such as ‘CGPA’, “Interactiveness’, ‘Practical Knowledge’ and ‘Communication Skills’ as shown in Table 8.1. The target variable is Job Offer which is classified as Yes or No for a candidate student. Step 1: Compute the prior probability for the target feature ‘Job Offer’. The target feature ‘Job ‘Offer’ has two classes, ‘Yes’ and ‘No’. Itis a binary classification problem. Given a student instance, we need to classify whether ‘Job Offer = Yes’ or ‘Job Offer = No’. From the training dataset, we observe that the frequency or the number of instances with ‘Job Offer = Yes’ is 7 and ‘Job Offer = No’ is 3. The prior probability for the target feature is calculated by dividing the number of instances belonging to a particular target class by the total number of instances. Hence, the prior probability for ‘Job Offer = Yes’ is 7/10 and ‘Job Offer = No’ is 3/10 as shown in Table 82. Bayesian Learning + 239 Table 8.2: Frequency Matrix and Prior Probability of Job Offer Job Offer Classes No, of Instances Eee Yes 7 P (Job Offer = Yes) = 710 [Ne 3 P (Job Offer = No) = 3/10 | step 2: Compute Frequency matrix and Likelihood Probability for each of the feature. step 2(a): Feature ~ CGPA Table 8.3 shows the frequency matrix for the feature CGPA. Table 8.3: Frequency Matrix of CGPA Ey orcs roe 29 3 1 28 4 0 8 0 2 Total 7 3 Table 84 shows how the likelihood probability is calculated for CGPA using conditional probability Table 8.4: Likelihood Probability of CGPA enon) P (CGPA.29 | Job Offer = No) = 1/3 [P (CGPA 28 | Job Offer = No) = 0/3 P (CGPA & | Job Offer = No) = 2/8 os Bens 29_| P(CGPA29 | Job Offer 28 __| P (CGPA28 | Job Offer 1 (Continued) 296 + Machine Learning CREE as Wq> Wy. 08 x Error, x O, =0.2 +08 x-0.0101 x0 14+ 0.8 x Error, x O, =-03 +08 « 0.005 x0 We | 104= wy + 08 x Error, x O, 0.092 1408 x 0.0101 x1 w,+ 08 «Error, xO, 0.408 A +08 x 0,005 «1 Wy |0,= 1,408 xError,x 0, 0.232 0.3408 «0.141 x 0.599 W,,+ 08 x Error, x O, 0.287 =0.2+ 080.141 0769 Update bias using the below formula: Ped 0.192 02402 x -0.0101 0, | = 8+ax Error, 0.04 11 +08 0.008 re =-03+08%0181 Iteration 2 Now, with the updated weights and biases: 1. Calculate Input and Output in the Input Layer as shown in Table 10.13. Table 10.13: Net Input and Output Calculation 1 1 0 1 2. Calculate Net Input and Output in the Hidden Layer and Output Layer as shown in Table 10.14. 4 oe aptificiat Neural Networks 6 207 Table 10.14: Net Input and Output Calculation in the Hidden Layer and Output Layer Wy Hx yg NG X Wg HH Wy XB, 0.292 +1 x 0.208 + 0.0.24 1 x 0.092 +1 x 0.192 = 0.368 Wag FAX Wy AK, yt HG AXXO, 0.292 +1 x 0.404 +0 x-0.3 +1 x 0.404 + 1 x 0.104 = 1.204 1 Wy + O,% ty + 4% 8, 1, = 0.591 x -0.232 + 0.7692 « 0.287 + 1 x -0.187 = -0.926 The output we receive in the network at node 7 is 0.474, Error =1 - 0.474 = 0.526 Now, when we compare the error we get in the previous iteration and in the current iteration, it is visible that the network has learnt and reduced the error by 0.055. Error is reduced by 0.055: 0.581 ~0.526. ‘Thus, the training is continued for a predefined number of epochs or until the training error is reduced below a threshold value. St“ 10.7 RADIAL BASIS FUNCTION NEURAL NETWORK Radial Basis Function Neural Network (RBFNN) was introduced by Broomhead and Lowe in 1988. It is a type of Multi Layer Perceptron which has one input layer, one output layer and with strictly one hidden layer. The hidden layer uses a non-linear radial basis function as the activation function, which converts the input parameters into high dimension space which is then fed into the network to linearly separate the problem. An XOR function is not linearly separable and requires at least one hidden layer to classify it. The RBFNN uses the hidden layer to derive the feature vector ‘whose dimension is increased in space. This neural network is useful for interpolation, function approximation, time series prediction, classification and system control. Typical Radial Basis Functions (RBF) are: The Gaussian RBF which monotonically decreases with distance from the centre. (10.17) where, cis the centre and r is the radius. A Multiquadric RBF which monotonically increases with distance from the centre. (10.18) RBFNN architecture includes: 1, An input layer that feeds the input vector of n-dimension to the network (yxy ww, 2. A hidden layer that comprises ‘m’ non-linear radial basis function neurons where m2 n. The hidden layer implements the Radial Basis Function called Gaussian function. The output of a hidden layer neuron for an input vector x is given as in Eq. (10.17): ni H(z) =€ where, xis the input vector, cis the centre and r is the radius. 298 © Machine Learning Each RBF neuron in the hidden layer compares the input vector with the centre of the neuron which is a bell curve and outputs a similarity value between 0 and 1. If the input is equal to the neuron centre, then the output is 1 but as the difference increases, the activation value or the output of the neuron falls off exponentially towards 0. 3. An output layer that computes the linear weighted sum of the output of each neuron from the hidden layer neurons. Fay= En w(x) (10.19) where, " tw,is the weight in the link from the Hidden Layer neuron i to the Output Layer. H(q) is the output of a Hidden Layer neuron i for an input vector x. The architecture of a Radial Basis Function Neural Network is shown in Figure 10.12. Input layer Single hidden Output layer layer Figure 10.12: Architecture of RBFNN Training or learning with RBFNN is very fast and the neural network is very good at interpolation. Rte OR eset oc uretCui cre tne Input: Input vector (2, 23 ou» 2,) Output: Y, Assign random weights for every connection from the Hidden layer to the Output layer in the network in the range [-1, +]. Forward Phase: Step 1: Calculate Input and Output in the Input Layer: (Input layer is a direct transfer function, where the output of the node equals the input). Input at Node i ‘I; in the Input Layer is 1, where, xis the input received at Node i. (Continued) —_ ] So Artificial Neural Networks + 299 Output at Node i ‘Step 2: For each node j in the Hidden Layer, find the centre/receptor c and the variance r. Define hidden layer neurons with Gaussian RBF whose output is: Hx) where, x is the input, ¢, is the centre and r is the radius. ‘Compute (x ~ ¢)} applying Euclidean distance measure between x and c, Step 3: For each node k in the Output Layer, compute linear weighted sum of the output of each neuron k from the hidden layer neurons j. F@)= 37, 0H) where, w, is the weight in the link from the Hidden Layer neuron j to the Output Layer neuron k. HQ) is the output of a Hidden Layer neuron j for an input vector x. Backward Phase: Step 1: Train the Hidden layer using Back propagation. Step 2: Update the weights between the Hidden layer and Output layer. Consider the XOR Boolean function that has 4 patterns (0, 0) (0, 1) (1, 0) and Ho) Input layer Hidden layer ‘Output layer Figure 10.13: Example RBFNN Construct a RBFNN as shown in Figure 10.13 that classifies the input pattern: @H>0 @N>1 G1 ayo 300 + Machine Learning Solution: Define 4 hidden layer neurons with Gaussian RBF: ea H@) seat H,@) * je2=(0,1) pers Ha)=¢ * 5c3=(1,0) test H,@)=¢ " ;e4=(1,1) ‘For input pattern (0, 0) For input pattern (0, 1) Distance squared of x from cl = (0,0) =@-0F +(0-0'=0 te H@=e 7 =¢? =10 Distance squared of x from c2 = (0,1) =@-0F+@-1=1 seak H@=e ? =6? =06 Distance squared of x from c3 = (1, 0) = (0-17 + @-07=1 H@jwe * =6? =06 Distance squared of x from cf = (1, 1) = (0-17 + -17=2 seh H@=¢ 7 =e? =04 Et, wH, (x) =-08x10+09 x06+09%06+ Distance squared of x from cl = (0,0) = (0-07 + (1-0F=1 Hee wet n06 Distance squared of x from 2 = (0,1) =(0-07+ (1-170 seat H=e 7 =e? =10 Distance squared of x from c3 = (1,0) = (0-17 + (1-0F=2 ot Hyay=e =04 Distance squared of x from ¢4 = (1, 1) =(0-17+(1-1f=1 0? Ho=e =06 ZN, WH, (2) =-08%06 +09 %10409 x 04+ Distance squared of x from c3 = (1, 0) =(1-17+@-07=0 oe Hy)=e set =10 Distance squared of x from c4 = (1,1) = (1-1 + 0-121 sth H(=e 7 =? =06 EN, w\H (x) =-08x06-+09x04+09% 10+ ~08x06=03 08 x 0.4 =-0.04 0.8 x 0.6 =0.3, For input pattern (1, 0) For input pattern (1, 1) Distance squared of x from cl = (0, 0) Distance squared of x from cl = (0, 0) = (1-07 + @-07F=1 = (1-0 + (1-07=2 seat 4 weak H()=¢ 7 =e? =06 H@)=e * =e? =04 Distance squared of x from c2 = (0,1) Distance squared of x from c2 = (0,1) = (1-07 +@-17=2 =(-oF + (1-171 seh seat a H@=e * =62=04 H(x)=e "=e? =06 Distance squared of x from ¢3 = (1, 0) = (1-14 (1-0F=1 oF 4 Hae * =6? =06 Distance squared of x from c4 = (1, 1) = (1-14 (1-17=0 ho H@=e 7 =6? =10 Ei, w)H,(x) =-08 x04+09 x 0.6 +09 x 06+ -0.8 x1.0=-0.04 q Airtifictat Neurat Networks 302 Table 10.15 shows the obtained values during the calculation done in the forward phase. Table 10.15: Forward Phase Calculation ce RBF Networks are generally trained to determine the following parameters: 1, The number of neurons in the hidden layer 2. The center of each hidden-layer RBF neuron 3. The radius or variance of each RBF function 4. The weights assigned from the Hidden layer to the Output layer for the summation function Different approaches are followed to determine the centres for the Hidden layer RBF neurons, comprising: 1. Random selection of fixed cluster centres 2. Self-organized selection of centres using K-means clustering 3. Supervised selection of centres 10.8 SELF-ORGANIZING FEATURE MAP Self-Organizing Feature Map (SOFM) is a special type of Feed Forward Artificial Neural Network developed by Dr Teuvo Kohonen in 1982. Kohonen network is a competitive learning network or also called as adaptive learning network. SOM is an unsupervised learning model that clusters data by mapping a high-dimensional data into a two-dimensional map (neurons) or plane. The model learns to cluster or self organize a high-dimensional data without knowing the class membership of the input data, and hence the name self-organizing nodes. These self-organizing nodes are also called as feature maps. The mapping is based on the relative distance or similarity between the points and the points that are near to each other in the input space are mapped to nearby output map units in the SOFM. Network Architecture and Operations ‘The network architecture consists of only two layers called the Input layer and the Output layer, and there are no Hidden layers. The number of units in the Input layer is based on the length of the input samples which is a vector of length ‘n’. Each connection from the Input units in the Input layer to the output units in the Output layer is assigned with random weights. There is one weight vector of length ‘n’ associated with each output unit. Output units have intra layer connections with no weights assigned between these connections but used for updating the weights. The network architecture of SOFM is shown in Figure 10.14.

You might also like