ARTIFICIAL INTELLIGENCE

Comprehensive Study Notes

Covering Fundamentals · Problem Solving · Python · AI Programming

MODULES COVERED
Module 1: Fundamentals of Artificial Intelligence

Module 2: Problem Solving Techniques

Module 3: Python Programming Environment

Module 4: Developing AI Programs Using Python

MODULE 1: FUNDAMENTALS OF ARTIFICIAL INTELLIGENCE

1.1 Definition of Artificial Intelligence

Artificial Intelligence (AI) is the branch of computer science concerned with building smart
machines capable of performing tasks that typically require human intelligence. The term
was coined by John McCarthy in 1956.
Artificial Intelligence (AI) refers to the ability of machines or computer systems to perform
tasks that normally require human intelligence.

Key characteristics of AI:

• Learning from experience (Machine Learning)

• Reasoning and decision making

• Problem solving

• Perception (vision, speech recognition)

• Natural language understanding

Formal definition:
AI is a branch of computer science concerned with building intelligent agents that can
perceive their environment and take actions to achieve specific goals.

Artificial intelligence (AI) is a branch of computer science that develops systems capable of
simulating human cognitive processes—such as learning, reasoning, problem-solving,
perception, and decision-making—to perform tasks independently or with minimal
oversight.

Key definitions from leading authorities:

• John McCarthy (1956): AI is the science and engineering of making intelligent machines.
• Marvin Minsky: AI is the science of making machines do things that would require
intelligence if done by humans.
• Russell & Norvig: AI is the study of agents that perceive their environment and take
actions that maximize their chances of achieving their goals.
• Nils Nilsson: AI is concerned with the intelligent behavior in artifacts.

Core Goals of AI
• Simulate human reasoning and problem solving
• Build systems that learn from data and improve over time
• Automate complex tasks that require judgment and adaptability
• Enable natural interaction between humans and machines
💡 Key Insight
 AI is not just about mimicking humans — it's about creating systems that act rationally and
achieve goals effectively, sometimes surpassing human performance.

1.2 History of Artificial Intelligence

The development of AI spans over eight decades and is marked by periods of excitement ('AI
summers') and disillusionment ('AI winters').

Timeline of AI History
Period Era Key Developments
1940s–1950s Foundations Alan Turing proposes the Turing
Test (1950). Warren McCulloch
& Walter Pitts model artificial
neurons (1943). Norbert Wiener
develops Cybernetics. The
modern concept of AI began with
Alan Turing, who asked: Can
machines think?
He proposed the Turing Test as
a way to measure machine
intelligence.
1956 Birth of AI John McCarthy organizes the
Dartmouth Conference — AI is
officially named as a field.
McCarthy, Minsky, Shannon, and
Simon attend.
1956–1974 First AI Summer Early programs like ELIZA
(natural language), DENDRAL
(chemistry), and GPS (General
Problem Solver) are developed.
Optimism is high. Notable
systems:
ELIZA (a simple chatbot)
Shakey the Robot (early mobile
robot)
Limitations in computing power
and data soon became clear.
1974–1980 First AI Winter Funding is cut due to unmet
expectations. Limitations of early
systems become apparent.
DARPA withdraws support.
1980–1987 Expert Systems Boom Rule-based expert systems (e.g.,
MYCIN for medical diagnosis,
XCON for computer
configuration) dominate.
Commercial interest returns.
1987–1993 Second AI Winter Expert systems prove brittle and
expensive to maintain. Funding
dries up again.
1993–2011 AI Resurgence Machine Learning gains traction.
Deep Blue defeats Kasparov in
chess (1997). Statistical
approaches replace rule-based
systems. AI shifted toward
 machine learning—systems
that learn from data.
Key milestone: IBM’s Deep Blue
defeated chess champion Garry
Kasparov in 1997.
Advances in:
• Speech recognition
• Computer vision
• Data-driven algorithms

2012–Present Deep Learning Era AlexNet wins ImageNet (2012).

GPT, BERT, and large language
models emerge. AlphaGo defeats
world champion Go player
(2016). ChatGPT launched
(2022).

1.3 Foundations of Artificial Intelligence

AI draws from multiple disciplines. Understanding these foundations is essential to
appreciating how AI systems are built.

1. Mathematics & Logic

• Formal logic: propositional and predicate logic for knowledge representation
• Probability theory: for reasoning under uncertainty (Bayesian inference)
Inferencing: Connecting clues to reach a conclusion.
• Linear algebra: underpins neural networks and machine learning
• Calculus: optimization using gradient descent
• Statistics: modeling distributions, hypothesis testing. (untested explanation, educated
guess)

2. Computer Science
• Algorithms and data structures for search and optimization
• Automata theory and computability
• Programming languages: Python, Prolog, Lisp
• Databases for knowledge storage

3. Cognitive Science & Psychology

Cognitive refers to the mental processes of acquiring knowledge and understanding through
thought, experience, and senses, encompassing memory, attention, problem-solving, and
decision-making.

Psychology refers to the study of how the brain works, how people think, and why they act in
certain ways.
• Models of human memory, learning, and reasoning
• Gestalt psychology influencing pattern recognition: . Its core philosophy, often
summarized as "the whole is different than the sum of its parts," posits that the brain
 naturally organizes sensory information into patterns, shapes, and meaningful wholes,
rather than collecting isolated elements.
• Cognitive architectures like:
ACT-R (Adaptive Control of Thought-Rational): ACT-R is strongly rooted in cognitive
psychology (modeling human behavior)
SOAR (Security Orchestration, Automation, and Response): rooted in artificial
intelligence (building intelligent agents)

4. Neuroscience

Neuroscience is the scientific study of the nervous system, including the brain, spinal cord,
and neural networks, spanning from molecular levels to behavior. It explores the structure,
function, development, and pathology of the nervous system, aiming to understand how it
regulates emotions, thoughts, behaviors, and body functions.

• Biological neural networks inspire artificial neural networks

• Perceptron model (Rosenblatt, 1958) based on neuron firing
• Backpropagation mimics synaptic weight adjustment

5. Philosophy

is the systematic study of ideas and issues, a reasoned pursuit of fundamental truths, a quest
for a comprehensive understanding of the world.
• Epistemology: the nature of knowledge and belief
• Ethics: AI safety, bias, fairness
• Philosophy of mind: can machines be conscious?
• Searle's Chinese Room argument challenges strong AI
6. Linguistics

the scientific study of language, focusing on its structure, use, development, and cognitive
basis, rather than just learning to speak multiple languages

Phonetics & Phonology: The study of speech sounds, their physical production, and
patterns.

Morphology: How words are formed and structured.

Natural Language Processing (NLP) relies on linguistics
• Syntax, semantics, pragmatics, and discourse analysis
• Chomsky's generative grammar influenced early AI language models
7. Engineering
Engineering is the practical application of science, mathematics, and design principles to
solve technical problems, innovate, and improve systems and infrastructure.
• Hardware design
• Robotics systems
8. Decision Theory
• Choosing best action under uncertainty
• Cost-benefit analysis.

1.4 Applications of Artificial Intelligence

AI applications span virtually every industry and domain of human activity.
Applications of Artificial Intelligence

1. Expert Systems
•
 Definition
An expert system is an AI program that simulates the decision-making ability of a human
expert in a specific domain.
Components:
• Knowledge Base → Contains facts and rules
• Inference Engine → Applies logical rules to data
• User Interface → Allows interaction with users
How it Works:
1. User inputs a problem
2. System searches knowledge base
3. Inference engine applies rules
4. System produces a solution
Applications:
• Medical diagnosis (e.g., disease identification)
• Financial advisory systems
• Technical troubleshooting
Advantages:
• Consistent decisions
• Available 24/7
• Reduces need for human experts
Limitations:
• Expensive to develop
• Limited to specific knowledge domain

2. Machine Learning (ML)

 Definition
Machine Learning is a subset of AI that enables systems to learn from data and improve
performance without being explicitly programmed.
An artificial intelligence (AI) agent refers to a system or program that is capable of
autonomously performing tasks on behalf of a user or another system.
Types:
• Supervised Learning → Supervised machine learning is one of the types of
machine learning where the algorithms are trained by labelled data, and every
training example contains the input data together with the right output. Through this
process, the model becomes capable of mapping inputs to the correct output.
Supervised learning as the name suggests, works like
a teacher or supervisor guiding the machine. In this approach we teach or train the
machine using the labelled data(correct answers or classifications) which means each
input has the correct output in the form of answer or category attached to it. After
that machine is provided with a new set of examples (data) so that it can analyses the
training data and produces a correct outcome from labeled data.
For example, a labeled dataset of images of Elephant, Camel and Cow would have
each image tagged with either "Elephant", "Camel" or "Cow."
 example
Imagine we have a basket full of different fruits that we want the machine to identify.
The machine first looks at the image of a fruit and extracts features like its shape,
color and texture. Then it compares these features to the fruits it has already learned
during training. If the new fruit’s features closely match those of an apple, the
machine will predict that the fruit is an apple.
For example, suppose we train the machine by showing it fruits one by one:
If the fruit is round, has a small depression at the top and is red, it is labeled as an
Apple.
If the fruit is long, curved and greenish-yellow, it is labeled as a Banana.
Now after this training, if we give the machine a new fruit (say a banana) from the
basket and ask it to identify it, the machine will use what it has learned during
training. It will analyze the shape and color of the new fruit and classify it as a Banana
placing it in the correct category. In this way, the machine learns from the training
data (the basket with labeled fruits) and applies that knowledge to recognize new,
unseen fruits.

Categories of Supervised Machine Learning

• There are mainly two types of Supervised Machine learning. These are:
Classification: Classification is a type of supervised machine learning where we teach
a computer to sort data into specific groups or categories. To do this, we use a special
type of data called ‘labelled data,’ where each piece of data is already tagged with the
correct category it belongs to. The computer uses this labelled data to learn and make
predictions about new, unseen data. The most often used algorithms for classification
are decision trees, logistic regression, and support vector machines as well as neural
 networks. Examples of the use of classification include the ability of a system to filter
out email spam.
Regression: Regression is another broad subheading of supervised machine learning
algorithms that target predictive analytics. While, as in the classification method, the
model does not involve nominal outcome variables, regression models are trained
with labelled data to identify a dependency between input parameters and a non-
discrete target variable. Popular algorithms are linear regression technique,
polynomial regression technique, and support vector regression technique.
Regression is applied quite often in many fields, for instance, it is used for stock price
prediction, forecasting of sales and, most importantly, estimation of real estate prices,
as the results are expected to be numerically accurate.
Common Algorithms
Some Common Algorithms of Supervised Machine Learning are:
• Logistic Regression
• Decision Trees
• Random Forests
• Support Vector Machines (SVM)
• Neural Networks
Applications of Supervised Learning
Some of the applications of supervised learning are:
• Email Spam Detection: In supervised learning methods, emails are classified as
spam or not spam to enhance the email filtering and users’ experience.
• Image Recognition: They are used for pattern recognition, object identification, faces
or scenes in images, with applications in biometrics/face recognition, self-driving
cars, medical image analysis, etc.
• Credit Scoring: Financial institutions use supervised learning to assess the
creditworthiness of applicants by predicting default risk based on historical data and
attributes.
• Customer Churn Prediction: Supervised learning is adopted by business
organizations to identify customers likely to leave their services so that actions to
retain them can be taken.
• Disease Diagnosis: Supervised learning helps diagnose diseases based on a patient’s
symptoms and past medical records, detect early disease, and find the most
appropriate treatment procedure.
• Unsupervised Learning → Finds patterns in unlabeled data.
Unsupervised learning is one of the most used machine learning types. It is a process
of constructing a model to recognize patterns from the data set without any input or
output variable in mind. It lacks the labeled training instances as does the supervised
learning, but this way, the model is free to discover the underlying structure of the
data on its own.
Unsupervised Learning is a type of machine learning where the model works without
labelled data. It learns patterns on its own by grouping similar data points or finding
hidden structures without any human intervention.
It is used for tasks like clustering, dimensionality reduction and Association Rule
Learning.
Helps identify hidden patterns in data
Useful for grouping, compression and anomaly detect
 The image shows set of animals like elephants, camels and cows that represents raw
data that the unsupervised learning algorithm will process.
The "Interpretation" stage signifies that the algorithm doesn't have predefined labels
or categories for the data. It groups the data based on inherent patterns.
An algorithm represents unsupervised learning process that helps to identify patterns
in the data.
The processing stage shows the algorithm working on the data.
The output shows the results of the unsupervised learning process. In this case, the
algorithm might have grouped the animals into clusters based on their species
(elephants, camels, cows).

Categories of Unsupervised Machine Learning

Unsupervised Machine learning is categorized into two types:
• Clustering: Clustering is a type of unsupervised machine learning where data is
partitioned into categories according to similarities within data points. In this case,
the major objective is to find a set of structures that are embedded in the data as they
are without any predetermined categorizations. Some of the most commonly used
clustering algorithms are the K-means algorithm, hierarchical clustering algorithm,
and DBSCAN. Thus, clustering is an industry-standard technology implemented in
different fields like consumer segmentation, storing data in a structured manner, and
image and music data in which complex information is divided into simple chunks for
easy understanding.
• Association Rules: Association rules is another category of machine learning that has
unsupervised learning, which primarily aims at finding interactions in variables
between large data sets. This approach seeks to establish relationships or a match,
basically in the form of a rule such as “If A then B”. This is done using techniques like
Apriori and FP-Growth. The goal is to understand how things are related, and it’s
 often used in retail to figure out which products are usually bought together. This
helps stores arrange their products in a way that makes sense to customers and
makes them happy.

Applications of Unsupervised Learning

Some of the applications of unsupervised learning are:
• Customer Segmentation: Companies apply clustering algorithms to sort out the
buyers including their buying habits, age, and preferences. With this, one can focus on
targeted marketing strategies.
• Anomaly Detection: It also focuses on finding unusual patterns in the data, which is
important in fraud detection, secure networking, and monitoring the industrial
system for any sign of failure.
• Market Basket Analysis: Each type of association rule contains different principles
because retailers use association rules to find out the frequent purchasing of products
and assist in inventory control and sales promotions.
• Recommendation Systems: Unsupervised learning through clustering and
association enables the creation of individual user solutions for the platform and,
ultimately, increases user interaction on platforms such as streaming or e-commerce
platforms.
Semi-Supervised Learning

Semi-supervised learning can be defined as a type of machine learning that shares features
with both supervised and unsupervised learning. In this method, the training is performed on
a small amount of labelled data collected through the proposed label propagation method and
a large number of unlabeled data. This is especially beneficial when labelling data is either
expensive or requires a lot of time.

The components of the model improve the learning from the labelled examples and, in the
same process, enhance its ability to generalize based on the information provided in the
unlabeled data samples. Other areas that semi-supervised learning can be used include image
classification, text categorization, and speech recognition, where it is difficult to get labelled
data.

Where to Use Semi-Supervised Learning

Use semi-supervised learning when the labels are on the expensive side, but the unlabeled
data can be easily acquired. It is common in medical images, fraud cases, customer support
tickets, or speech and text data. You would label a smaller set by experts and then use the
larger unlabeled set to train better features, leading to better accuracy. Methods like pseudo-
labeling and consistency checks can help improve performance with less manual work.
It i mostly Useful when you want quick progress but have budget constraints for full labeling.

It works best when unlabeled data looks like the labeled set. If the data shifts, the model may
learn the wrong patterns and must be monitored.

Reinforcement Learning → Learns through rewards and penalties

Reinforcement Learning is one of the types of machine learning. It is a way for AI to learn by
trial and error. It interacts with its environment, tries different actions, and gets rewards or
penalties. Over time, the AI figures out what works best and adjusts its behavior to achieve its
goals.

This type of learning is useful in situations like playing games, controlling robots, or
running automated systems, where the AI needs to make smart decisions and adapt to its
surroundings.
Categories of Reinforcement Learning
Reinforcement learning can be categorized into:
• Positive Reinforcement Learning: It is a way to teach AI by rewarding good
behavior. When the AI does something correctly, it gets a reward, which strengthens
the connection between the action and the desired outcome. This makes the AI more
likely to repeat the same behavior in the future because it learns that it leads to a
positive result. Some of the most used examples include offering a treat to the pet
after obedience and offering the player a bonus, for example, after scoring a goal in a
 game to ensure that they continue doing the right thing or, in this case, scoring
positive results in the game.
• Negative Reinforcement Learning: Negative Reinforcement Learning is when an AI
learns to do something because it stops a bad or uncomfortable thing from happening.
For example, a student might read to avoid getting a bad grade, or a gamer might
playto avoid losing. This type of learning doesn’t punish the AI but instead encourages
it to repeat the behavior by removing the unpleasantness. It’s like the AI learns to do
something because it makes a bad feeling go away.
Applications of Reinforcement Learning
Some of the applications of reinforcement learning are:
• Autonomous Vehicles: Self-driving cars utilize reinforcement learning so as to move
around and respond to their surroundings. It enables the vehicle to figure out the best
strategy through simulation of other driving situations and in response to feedback in
the form of signals from sensors and the surroundings.
• Robotics: Reinforcement learning opens the way to the acquisition of complex tasks
by robots, for example, through interactive learning as a process based on practice.
This approach is used in robotic grasping, manipulation, and navigation to enable the
robot to increase its performance in real-time.
• Game Playing: Reinforcement learning has been highly successful in game playing
and is especially evident in games such as AlphaGo, where AI programs carry out
many games to come up with a strategy that would better that of human beings.
• Healthcare: Reinforcement learning frameworks work to enhance the plan of
treatment with respect to the patient responses over time so that it may enhance
patient health outcomes in the case of chronic disease management and clinical
decisions.
• Finance and Trading: RL is used in finance for algorithmic trading, portfolio
optimization, and for evaluating risks. These models learn to buy or sell in accordance
with market conditions to maximize profit and, at the same time, minimize the risks.
3. Natural Language Processing (NLP)
•

•
 Definition
Natural Language Processing enables computers to understand, interpret, and generate
human language.
Components:
• Syntax analysis
• Semantic analysis
• Speech recognition
Applications:
• Chatbots
• Language translation
• Voice assistants (e.g., Siri)
• Sentiment analysis
Advantages:
• Improves human-computer interaction
• Automates communication
Limitations:
• Language ambiguity
• Context understanding challenges.

4. Gaming
 Definition
AI in gaming is used to create intelligent behaviors in non-player characters (NPCs) and
simulate realistic environments.
 Applications:
• NPC decision making
• Pathfinding algorithms
• Strategy development
Example:
• Deep Blue defeating a world chess champion
Advantages:
• Realistic gameplay
• Adaptive difficulty levels
Limitations:
• High computational cost

5. Artificial Neural Networks (ANN)

•

Definition
Artificial Neural Networks are computing systems inspired by the human brain, designed
to recognize patterns and solve complex problems.
Structure:
• Input Layer
• Hidden Layers
• Output Layer
 Applications:
• Image recognition
• Speech recognition
• Medical diagnosis
Key Concept:
• Uses weighted connections and activation functions
Advantages:
• High accuracy
• Learns complex patterns
Limitations:
• Requires large data
• Difficult to interpret (black box problem)

6. Computer Vision
•
 Definition
Computer Vision enables machines to interpret and understand visual information from
the world.
Applications:
• Facial recognition
• Object detection
• Medical image analysis
• Autonomous vehicles
Techniques:
• Image processing
• Pattern recognition
• Deep learning
Advantages:
• High-speed image analysis
• Improves automation
Limitations:
• Sensitive to image quality
• Requires large datasets

1.5 Intelligent Agents

An intelligent agent is any entity that perceives its environment through sensors and acts
upon that environment through actuators to achieve goals. This is the central concept in
modern AI (Russell & Norvig).

Components of an Intelligent Agent

• Percepts: inputs from the environment (e.g., camera image, keyboard input, sensor
reading)
• Actions: outputs that affect the environment (e.g., motor commands, text output)
• Agent Function: maps percept sequences to actions
• Agent Program: the implementation of the agent function
• Performance Measure: evaluates how well the agent is achieving its goals

Types of Agents (by capability)

• Simple Reflex Agent: Acts only on current percept. Uses condition-action rules. No
memory. Example: thermostat.
• Model-Based Reflex Agent: Maintains internal state (model of world). Handles partially
observable environments. Example: robot vacuum.
• Goal-Based Agent: Has explicit goals and takes actions to achieve them. Uses search and
planning. Example: GPS navigation.
• Utility-Based Agent: Maximizes a utility function (degree of happiness). Makes trade-offs
between competing goals. Example: self-driving car.
• Learning Agent: Improves performance over time through experience. Has a learning
element, performance element, critic, and problem generator. Example: AlphaGo.

Environments (PEAS Framework)

Agents are designed for specific environments. PEAS stands for:
• Performance measure: criteria for evaluating success
• Environment: where the agent operates
• Actuators: how the agent acts
• Sensors: how the agent perceives
Example — Self-driving car:
• P: Safe driving, legal compliance, comfort, fuel efficiency
• E: Roads, other cars, pedestrians, weather, traffic lights
• A: Steering, accelerator, brakes, horn, display
• S: Cameras, GPS, LIDAR, speedometer, radar

Environment Properties
• Fully vs Partially Observable: Can the agent see the complete state?
• Deterministic vs Stochastic: Does the same action always produce the same result?
• Episodic vs Sequential: Are actions independent or do they depend on past actions?
• Static vs Dynamic: Does the environment change while the agent is thinking?
• Discrete vs Continuous: Are states and actions countable or real-valued?
• Single vs Multi-Agent: Is the agent alone or competing/cooperating with others?

1.6 AI Applications in Real Life

Beyond theoretical applications, AI is already embedded in our everyday experiences.
• Smartphone Voice Assistants: Siri (Apple), Google Assistant, and Alexa use NLP, speech
recognition, and knowledge graphs to answer queries and control devices.
• Social Media Feeds: Facebook, Instagram, and TikTok use reinforcement learning to
curate personalized content feeds and advertisements.
• Email Spam Filters: Gmail's spam filter uses Naive Bayes and neural networks to classify
emails, achieving >99% accuracy.
• Streaming Recommendations: Netflix and Spotify use collaborative filtering and deep
learning to recommend content, driving 80% of Netflix viewing.
• Online Banking Fraud Detection: Banks use anomaly detection algorithms to flag
suspicious transactions in real time, preventing billions in fraud annually.
• Medical Imaging: Google's DeepMind detects eye diseases from retinal scans. FDA-
approved AI tools diagnose diabetic retinopathy.
• Search Engines: Google uses BERT and MUM (large language models) to understand the
intent behind search queries.
• Autonomous Vehicles: Tesla's Autopilot uses computer vision and deep learning to
enable semi-autonomous driving on highways.
• Smart Home Devices: Nest thermostats learn schedules and preferences over time to
optimize energy usage automatically.
• Agricultural Tech: Drones with computer vision identify crop diseases and pest
infestations, enabling targeted intervention.
MODULE 2: PROBLEM SOLVING TECHNIQUES

2.1 Logical Operators

Logic is the foundation of AI reasoning. Logical operators are used to combine propositions
and derive conclusions.

Basic Logical Operators

Operator Meaning Example
AND (∧ / Conjunction) True only if both operands are P ∧ Q: 'It is raining AND cold'
true.
OR (∨ / Disjunction) True if at least one operand is P ∨ Q: 'It is raining OR sunny'
true.
NOT (¬ / Negation) Inverts the truth value. ¬P: 'It is NOT raining'
IMPLIES (→ / Implication) If P then Q. False only when P is P → Q: 'If it rains, the ground is
true and Q is false. wet'
BICONDITIONAL (↔) True when both sides have the P ↔ Q: 'P if and only if Q'
same truth value.
XOR (⊕ / Exclusive OR) True when exactly one operand P ⊕ Q: 'Either P or Q but not
is true. both'
NAND (↑) Negation of AND — false only ¬(P ∧ Q)
when both are true.
NOR (↓) Negation of OR — true only ¬(P ∨ Q)
when both are false.

Truth Table for Key Operators

P Q P∧Q P∨Q ¬P P→Q P↔Q
True True True True False True True
True False False True False False False
False True False True True True False
False False False False True True True

Laws of Logic
• De Morgan's Laws: ¬(P∧Q) ≡ ¬P∨¬Q and ¬(P∨Q) ≡ ¬P∧¬Q
• Contrapositive: P→Q ≡ ¬Q→¬P
• Double Negation: ¬¬P ≡ P
• Commutativity: P∧Q ≡ Q∧P and P∨Q ≡ Q∨P
• Associativity: (P∧Q)∧R ≡ P∧(Q∧R)
• Distributivity: P∧(Q∨R) ≡ (P∧Q)∨(P∧R)

2.2 Propositional and Predicate Logic

Propositional Logic (PL)
Propositional logic (also called zeroth-order logic) deals with simple statements
(propositions) that are either true or false.
• A proposition is a declarative statement with a definite truth value.
• Propositions are represented by symbols: P, Q, R, ...
• Complex propositions are formed using logical operators
• Example: Let P = 'It is raining', Q = 'The road is wet'. Then P → Q means 'If it is raining, the
road is wet'.

Limitations of Propositional Logic

• Cannot express relationships between objects
• Cannot quantify over individuals ('All students are intelligent')
• Cannot express properties of objects

Predicate Logic (First-Order Logic — FOL)

Predicate logic extends propositional logic by introducing predicates, quantifiers, and
variables to reason about objects and their relationships.

Key Elements of Predicate Logic

• Constants: Specific individuals — John, 5, Paris
• Variables: Unspecified individuals — x, y, z
• Predicates: Properties or relationships — Smart(x), Loves(x,y)
• Functions: Map objects to objects — fatherOf(John) = Tom
• Quantifiers: Express 'for all' or 'there exists'

Quantifiers
• Universal Quantifier (∀): 'For all x'. Example: ∀x Student(x) → Works_Hard(x) — 'All
students work hard'
• Existential Quantifier (∃): 'There exists x'. Example: ∃x Loves(John, x) — 'John loves
someone'
• Nested Quantifiers: ∀x ∃y Loves(x, y) — 'Everyone loves someone'
💡 Propositional vs Predicate Logic
Propositional: 'Socrates is mortal' (P)
Predicate: Mortal(Socrates) — can express: ∀x Human(x) → Mortal(x); Human(Socrates) ⊢
Mortal(Socrates)

Inference Rules in Predicate Logic

• Universal Instantiation: From ∀x P(x), infer P(c) for any constant c
• Existential Instantiation: From ∃x P(x), infer P(c) for some new constant c
• Universal Generalization: From P(c) for any arbitrary c, infer ∀x P(x)
• Existential Generalization: From P(c), infer ∃x P(x)

2.3 Types of Inferencing

Inferencing is the process of deriving new conclusions from known facts. It is central to how
AI systems reason.

1. Deductive Inference
Derives specific conclusions from general rules. If the premises are true, the conclusion must
be true.
Form: Premise 1 + Premise 2 → Conclusion
All mammals breathe air. (General rule)
Dogs are mammals. (Specific fact)
∴ Dogs breathe air. (Conclusion — necessarily true)

• Used in: Expert systems, Prolog programming, theorem provers

• Advantage: Guarantees correct conclusions if premises are correct
• Limitation: Cannot generate new knowledge beyond what is in the premises

2. Inductive Inference
Derives general rules from specific observations. Conclusions are probable, not certain.
Observation 1: This crow is black.
Observation 2: That crow is black.
Observation 3: Every crow I've seen is black.
∴ All crows are black. (Probable but not guaranteed)

• Used in: Machine learning, statistical learning, scientific reasoning

• Advantage: Can generate new generalizations from data
• Limitation: Subject to counter-examples (a white crow disproves the conclusion)

3. Abductive Inference
Inference to the best explanation. Given an observation, finds the most plausible explanation.
Observation: The ground is wet.
Possible explanations: It rained, a pipe burst, sprinklers ran.
∴ The best explanation is: It rained. (Most plausible)

• Used in: Medical diagnosis, fault detection, plan recognition

• Advantage: Handles incomplete information; produces explanations
• Limitation: The 'best' explanation may not always be correct

4. Analogical Inference
Reasoning by drawing parallels between similar situations.
Situation A: Drug X cured disease Y in mice.
Analogy: Mice and humans have similar biology.
∴ Drug X might also cure disease Y in humans.

• Used in: Case-based reasoning, creative problem solving

5. Common Sense Reasoning

Draws on background knowledge that humans take for granted.
• Requires large knowledge bases (e.g., Cyc, ConceptNet)
• Challenge: Vast amount of implicit knowledge is hard to formalize

6. Non-Monotonic Reasoning
Conclusions can be revised when new information is added (unlike deductive logic where
conclusions are fixed).
• Default logic: Assume X unless there is evidence to the contrary
• Example: 'Tweety is a bird → Tweety can fly' (revised if Tweety is a penguin)

7. Probabilistic Inference
Handles uncertainty using probability theory.
• Bayes' Theorem: P(A|B) = P(B|A) × P(A) / P(B)
• Bayesian networks: graphical models of probabilistic relationships
• Used in: spam filters, medical diagnosis, speech recognition

2.4 Machine Learning — Definition

Machine Learning (ML) is a subset of AI in which systems automatically learn and improve
from experience (data) without being explicitly programmed. It focuses on developing
algorithms that can access data and use it to learn for themselves.
Arthur Samuel (1959): 'Machine Learning gives computers the ability to learn without being
explicitly programmed.'
Tom Mitchell (1997) formal definition: 'A computer program is said to learn from experience
E with respect to some class of tasks T and performance measure P, if its performance at
tasks in T, as measured by P, improves with experience E.'

Key ML Concepts
• Training Data: The dataset used to learn patterns
• Features: Input variables used for prediction
• Labels/Targets: Output values the model tries to predict
• Model: Mathematical function mapping inputs to outputs
• Training: Process of adjusting model parameters to minimize error
• Testing: Evaluating model on unseen data
• Overfitting: Model memorizes training data, fails on new data
• Underfitting: Model is too simple to capture the underlying pattern
• Generalization: Ability to perform well on unseen data
💡 ML vs Traditional Programming
Traditional Programming: Data + Rules → Output
Machine Learning: Data + Output → Rules (learns the rules automatically)

2.5 Types of Machine Learning

1. Supervised Learning
The algorithm learns from labeled training data — each example has an input and a known
correct output.
• Classification: Predicting a category. Examples: spam detection, image classification,
disease diagnosis
• Regression: Predicting a continuous value. Examples: house price prediction, stock
forecasting
• Common algorithms: Linear Regression, Logistic Regression, Decision Trees, Random
Forest, SVM, Neural Networks, K-Nearest Neighbours, Naive Bayes
Input: Features (size, location, rooms)
Label: House Price ($250,000)
Task: Learn the mapping Function → Price

2. Unsupervised Learning
The algorithm finds patterns in unlabeled data — no correct answers are provided.
• Clustering: Group similar data points. Examples: customer segmentation, document
grouping
• Dimensionality Reduction: Reduce features while preserving information. Examples: PCA,
t-SNE, autoencoders
• Association Rule Mining: Find co-occurrence patterns. Example: market basket analysis
('people who buy X also buy Y')
• Common algorithms: K-Means, DBSCAN, Hierarchical Clustering, PCA, Apriori,
Autoencoders

3. Reinforcement Learning (RL)

An agent learns by interacting with an environment, receiving rewards for good actions and
penalties for bad ones. The goal is to maximize cumulative reward.
• Key concepts: Agent, Environment, State, Action, Reward, Policy, Value Function
• Exploration vs Exploitation: Balance between trying new actions and using known good
ones
• Examples: Game playing (AlphaGo, Chess), robot locomotion, autonomous driving,
trading bots
• Algorithms: Q-Learning, SARSA, Deep Q-Networks (DQN), Policy Gradient, PPO
Agent takes Action → Environment gives Reward + New State
Agent updates Policy to maximize future rewards

4. Semi-Supervised Learning
Uses a small amount of labeled data combined with a large amount of unlabeled data.
• Practical when labeling data is expensive or time-consuming
• Examples: Medical imaging (few labeled scans, many unlabeled ones), web page
classification

5. Self-Supervised Learning
A special case where labels are automatically generated from the input data itself.
• The model predicts part of the input from other parts
• Examples: BERT predicts masked words; GPT predicts the next word
• Enables training on massive amounts of unlabeled data

6. Transfer Learning
A pre-trained model (trained on large datasets) is fine-tuned for a specific task.
• Reduces the need for large task-specific datasets
• Examples: Using ImageNet-trained ResNet for medical imaging; using GPT for customer
service

2.6 Applications of Different Types of Inferencing

Inference Type Applications
Deductive Expert systems for medical diagnosis (MYCIN);
Automated theorem proving; Prolog-based logic
programming; Tax calculation engines
Inductive Machine learning model training; Rule learning
from databases; Scientific hypothesis generation
Abductive Medical diagnostic systems (best explanation for
symptoms); Fault diagnosis in engineering; Natural
language understanding
Probabilistic Spam filtering (Naive Bayes); Weather forecasting;
Speech recognition; Bayesian networks for risk
assessment
Analogical Case-based reasoning in legal systems; Precedent-
based medical decisions; Creative design tools
Non-Monotonic Planning systems that revise plans when
circumstances change; Robot reasoning with
incomplete world knowledge
MODULE 3: PYTHON PROGRAMMING ENVIRONMENT

3.1 Installation of Python

Python is the dominant programming language for AI and machine learning due to its
simplicity, readability, and rich ecosystem of libraries.

Installation Steps
• Step 1: Go to [Link] and download the latest stable
Python 3.x release.
• Step 2: Run the installer. On Windows, check 'Add Python to PATH'.
• Step 3: Verify installation:
python --version
pip --version

• Step 4: Install a code editor — VS Code, PyCharm, or Jupyter Notebook.

• Step 5 (Recommended): Install Anaconda for data science (includes NumPy, Pandas,
SciPy, Jupyter).
# Install Anaconda, then create a virtual environment:
conda create -n ai_env python=3.11
conda activate ai_env

Setting Up a Virtual Environment (standard Python)

# Create environment
python -m venv myenv

# Activate (Windows)
myenv\Scripts\activate

# Activate (Mac/Linux)
source myenv/bin/activate

# Install packages
pip install numpy pandas scikit-learn matplotlib jupyter

Jupyter Notebook
• Interactive web-based IDE for writing and running Python code in cells
• Ideal for data exploration, visualization, and AI experiments
pip install jupyter
jupyter notebook # Launch in browser

3.2 Python Syntax

Python uses indentation (whitespace) to define code blocks — no curly braces. This enforces
clean, readable code.
Basic Syntax Rules
• No semicolons needed at end of statements
• Use 4 spaces for indentation (PEP 8 standard)
• Comments use # for single line, ''' or """ for multi-line
• Python is case-sensitive: Variable ≠ variable
• Statements end with a newline
# Single line comment
'''
Multi-line
comment / docstring
'''

# Print statement
print("Hello, Artificial Intelligence!")

# Variables (no declaration needed)

name = "Claude"
age = 25
pi = 3.14159
is_ai = True

# String formatting
print(f"Name: {name}, Age: {age}") # f-strings (Python 3.6+)
print("Pi is approximately %.2f" % pi)

# Multiple assignment
x, y, z = 1, 2, 3
a = b = c = 0 # All equal to 0

Python Keywords
Reserved words that cannot be used as variable names: if, else, elif, for, while, break,
continue, def, class, return, import, from, True, False, None, and, or, not, in, is, try, except,
finally, with, as, lambda, yield, pass, raise, del, global, nonlocal, assert

3.3 Data Types in Python

Built-in Data Types
# Integers
x = 42
big = 1_000_000 # Underscores for readability

# Floats
pi = 3.14159
sci = 2.5e-3 # Scientific notation

# Complex numbers
c = 3 + 4j

# Strings
s1 = 'Hello'
s2 = "World"
s3 = '''Multi
 line'''

# Booleans
is_true = True
is_false = False

# None type
nothing = None

# Type checking
print(type(42)) # <class 'int'>
print(type(3.14)) # <class 'float'>
print(type("hello")) # <class 'str'>
print(type(True)) # <class 'bool'>

Collections
# List — ordered, mutable, allows duplicates
fruits = ["apple", "banana", "cherry"]
[Link]("mango") # Add item
[Link]("banana") # Remove item
fruits[0] = "grape" # Modify item
print(fruits[1:3]) # Slicing

# Tuple — ordered, immutable

coords = (10.5, 20.3)
x, y = coords # Unpacking

# Set — unordered, unique elements

colors = {"red", "green", "blue"}
[Link]("yellow")
[Link]("red")
print("green" in colors) # Membership test

# Dictionary — key-value pairs

person = {"name": "Alice", "age": 30, "city": "Nairobi"}
print(person["name"]) # Access by key
person["email"] = "a@[Link]" # Add new key
[Link]("city") # Remove key
for key, value in [Link]():
print(f"{key}: {value}")

String Operations
s = "Hello, World!"

# Methods
print([Link]()) # HELLO, WORLD!
print([Link]()) # hello, world!
print([Link](", ")) # ['Hello', 'World!']
print([Link]("World", "AI")) # Hello, AI!
print([Link]()) # Remove whitespace
print(len(s)) # Length: 13
print([Link]("He")) # True
print([Link]("World")) # Index: 7

# String slicing
print(s[0:5]) # Hello
 print(s[-6:]) # World!
print(s[::-1]) # Reverse string

3.4 Control Structures in Python

Conditional Statements
score = 75

if score >= 90:

grade = "A"
elif score >= 80:
grade = "B"
elif score >= 70:
grade = "C"
elif score >= 60:
grade = "D"
else:
grade = "F"

print(f"Grade: {grade}")

# Ternary (one-line) if
result = "Pass" if score >= 50 else "Fail"

# Nested conditions
x, y = 5, 10
if x > 0:
if y > 0:
print("Both positive")

Loops
# For loop — iterating over sequence
for i in range(5): # 0, 1, 2, 3, 4
print(i)

for i in range(2, 10, 2): # start, stop, step

print(i) # 2, 4, 6, 8

# Iterating over list

fruits = ["apple", "banana", "cherry"]
for fruit in fruits:
print([Link]())

# Enumerate — get index and value

for i, fruit in enumerate(fruits):
print(f"{i}: {fruit}")

# Zip — iterate two lists together

names = ["Alice", "Bob", "Carol"]
scores = [85, 92, 78]
for name, score in zip(names, scores):
print(f"{name}: {score}")

# While loop
count = 0
 while count < 5:
print(count)
count += 1

# Break and continue

for i in range(10):
if i == 3:
continue # Skip 3
if i == 7:
break # Stop at 7
print(i)

# List comprehension (powerful Python feature)

squares = [x**2 for x in range(10)]
evens = [x for x in range(20) if x % 2 == 0]
matrix = [[i*j for j in range(3)] for i in range(3)]

Exception Handling
try:
num = int(input("Enter a number: "))
result = 100 / num
print(f"Result: {result}")
except ValueError:
print("Invalid input — not a number!")
except ZeroDivisionError:
print("Cannot divide by zero!")
except Exception as e:
print(f"Unexpected error: {e}")
else:
print("No errors occurred")
finally:
print("This always runs")

# Raising exceptions
def validate_age(age):
if age < 0:
raise ValueError("Age cannot be negative")
return age

3.5 Functions in Python

# Basic function
def greet(name):
"""Docstring: Greets a person by name."""
return f"Hello, {name}!"

print(greet("Alice"))

# Default parameters
def power(base, exponent=2):
return base ** exponent

print(power(3)) # 9
print(power(3, 3)) # 27

# *args — variable positional arguments

 def sum_all(*numbers):
return sum(numbers)

print(sum_all(1, 2, 3, 4, 5)) # 15

# **kwargs — variable keyword arguments

def describe(**info):
for key, value in [Link]():
print(f"{key}: {value}")

describe(name="Bob", age=30, city="Nairobi")

# Lambda functions (anonymous)

square = lambda x: x ** 2
multiply = lambda x, y: x * y
print(square(5)) # 25
print(multiply(3, 7)) # 21

# Higher-order functions
numbers = [1, 2, 3, 4, 5, 6]
evens = list(filter(lambda x: x % 2 == 0, numbers)) # [2,4,6]
squares = list(map(lambda x: x**2, numbers)) # [1,4,9,16,25,36]

from functools import reduce

product = reduce(lambda x, y: x*y, numbers) # 720

# Recursion
def factorial(n):
if n <= 1:
return 1
return n * factorial(n - 1)

print(factorial(5)) # 120

3.6 Object-Oriented Python

Python supports full OOP with classes, inheritance, polymorphism, and encapsulation.
# Class definition
class Animal:
# Class variable (shared by all instances)
kingdom = "Animalia"

# Constructor
def __init__(self, name, sound):
[Link] = name # Instance variable
[Link] = sound

# Instance method
def speak(self):
return f"{[Link]} says {[Link]}"

# String representation
def __str__(self):
return f"Animal({[Link]})"

# Class method
@classmethod
 def get_kingdom(cls):
return [Link]

# Static method
@staticmethod
def breathes():
return True

# Instantiation
dog = Animal("Dog", "Woof")
cat = Animal("Cat", "Meow")
print([Link]()) # Dog says Woof
print(Animal.get_kingdom()) # Animalia

# Inheritance
class Dog(Animal):
def __init__(self, name, breed):
super().__init__(name, "Woof") # Call parent constructor
[Link] = breed

# Method overriding (polymorphism)

def speak(self):
return f"{[Link]} the {[Link]} barks: Woof!"

def fetch(self):
return f"{[Link]} fetches the ball!"

rex = Dog("Rex", "German Shepherd")

print([Link]()) # Rex the German Shepherd barks: Woof!

# Multiple inheritance
class Pet:
def __init__(self, owner):
[Link] = owner
def get_owner(self):
return [Link]

class PetDog(Dog, Pet):

def __init__(self, name, breed, owner):
Dog.__init__(self, name, breed)
Pet.__init__(self, owner)

buddy = PetDog("Buddy", "Labrador", "Alice")

print(buddy.get_owner()) # Alice

# Encapsulation
class BankAccount:
def __init__(self, balance):
self.__balance = balance # Private attribute

def deposit(self, amount):

if amount > 0:
self.__balance += amount

def get_balance(self):
return self.__balance

@property
def balance(self): # Property decorator
return self.__balance
3.7 Scientific Modules in Python
NumPy — Numerical Computing
import numpy as np

# Arrays
a = [Link]([1, 2, 3, 4, 5])
b = [Link]([[1,2,3],[4,5,6]]) # 2D array

# Properties
print([Link]) # (5,)
print([Link]) # (2, 3)
print([Link]) # int64

# Array creation
zeros = [Link]((3, 3))
ones = [Link]((2, 4))
eye = [Link](3) # Identity matrix
rand = [Link](3, 3) # Random 3x3
seq = [Link](0, 10, 0.5) # Like range but supports floats
lin = [Link](0, 1, 50) # 50 evenly spaced points

# Operations (element-wise)
x = [Link]([1,2,3])
y = [Link]([4,5,6])
print(x + y) # [5 7 9]
print(x * y) # [4 10 18]
print([Link](x, y)) # Dot product: 32
print(x ** 2) # [1 4 9]

# Matrix operations
A = [Link]([[1,2],[3,4]])
print([Link](A)) # Determinant: -2
print([Link](A)) # Inverse
eigenvalues, eigenvectors = [Link](A)

# Statistical functions
data = [Link]([23, 45, 12, 67, 34, 89, 56])
print([Link](data)) # Mean
print([Link](data)) # Standard deviation
print([Link](data)) # Median
print([Link](data), [Link](data))

Pandas — Data Analysis

import pandas as pd

# Series
s = [Link]([10, 20, 30, 40], index=['a','b','c','d'])

# DataFrame
data = {
'Name': ['Alice', 'Bob', 'Carol', 'David'],
'Age': [25, 30, 35, 28],
'Score': [85, 92, 78, 95]
 }
df = [Link](data)

# Exploration
print([Link]()) # First 5 rows
print([Link]()) # Column types and non-null counts
print([Link]()) # Statistical summary

# Selecting data
print(df['Name']) # Select column
print(df[['Name', 'Score']]) # Multiple columns
print([Link][0]) # Row by label
print([Link][1:3]) # Rows by position
print(df[df['Score'] > 80]) # Filter rows

# Data manipulation
df['Grade'] = df['Score'].apply(lambda x: 'A' if x>=90 else 'B')
df.sort_values('Score', ascending=False, inplace=True)
[Link]('Age', axis=1, inplace=True)

# Handling missing data

[Link]() # Remove rows with NaN
[Link](0) # Fill NaN with 0

# Reading/writing files
# df = pd.read_csv('[Link]')
# df.to_csv('[Link]', index=False)
# df = pd.read_excel('[Link]')

Matplotlib — Visualization
import [Link] as plt
import numpy as np

# Line plot
x = [Link](0, 2*[Link], 100)
[Link](figsize=(10, 4))
[Link](x, [Link](x), label='sin(x)', color='blue')
[Link](x, [Link](x), label='cos(x)', color='red', linestyle=' --')
[Link]('Trigonometric Functions')
[Link]('x')
[Link]('y')
[Link]()
[Link](True)
[Link]()

# Bar chart
categories = ['A', 'B', 'C', 'D']
values = [23, 45, 12, 67]
[Link](categories, values, color=['blue','red','green','orange'])
[Link]('Category Values')
[Link]()

# Scatter plot
x = [Link](100)
y = 2*x + [Link](100)
[Link](x, y, alpha=0.5)
[Link]()
 # Histogram
data = [Link](0, 1, 1000)
[Link](data, bins=30, edgecolor='black')
[Link]()

# Subplots
fig, axes = [Link](2, 2, figsize=(10, 8))
axes[0,0].plot(x, [Link](x))
axes[0,1].plot(x, [Link](x))
plt.tight_layout()
[Link]()

SciPy — Scientific Computing

from scipy import stats, optimize, integrate

# Statistics
data = [2,4,4,4,5,5,7,9]
mean, std = [Link](data), [Link](data)
t_stat, p_value = stats.ttest_1samp(data, 5) # One-sample t-test

# Optimization
def f(x): return (x-3)**2 + 2
result = [Link](f, x0=0)
print(result.x) # [3.] — minimum at x=3

# Integration
result, error = [Link](lambda x: x**2, 0, 1)
print(result) # 0.333...
MODULE 4: DEVELOPING AI PROGRAMS USING PYTHON

4.1 Scikit-Learn (Sci-Kit Learn)

Scikit-learn is the most widely used Python library for classical machine learning. Built on
NumPy, SciPy, and matplotlib, it provides simple and efficient tools for data mining and data
analysis.

Key Features
• Consistent API: All models follow fit() → predict() pattern
• Wide algorithm coverage: classification, regression, clustering, dimensionality reduction
• Preprocessing tools: scaling, encoding, imputation
• Model evaluation: cross-validation, metrics, confusion matrices
• Pipeline support: chain preprocessing and models together

Installation
pip install scikit-learn

Scikit-Learn Ecosystem
Module Purpose
[Link] Built-in datasets (iris, digits, boston, breast_cancer,
wine)
[Link] Scaling (StandardScaler, MinMaxScaler), encoding
(LabelEncoder, OneHotEncoder)
sklearn.model_selection train_test_split, cross_val_score, GridSearchCV,
KFold
sklearn.linear_model LinearRegression, LogisticRegression, Ridge, Lasso
[Link] DecisionTreeClassifier, DecisionTreeRegressor
[Link] RandomForestClassifier,
GradientBoostingClassifier, AdaBoostClassifier
[Link] KNeighborsClassifier, KNeighborsRegressor
sklearn.naive_bayes GaussianNB, MultinomialNB, BernoulliNB,
ComplementNB
[Link] SVC, SVR, LinearSVC
[Link] KMeans, DBSCAN, AgglomerativeClustering
[Link] PCA, NMF, TruncatedSVD
[Link] accuracy_score, confusion_matrix,
classification_report, roc_auc_score,
mean_squared_error
[Link] Pipeline — chain steps together
sklearn.neural_network MLPClassifier, MLPRegressor

General Scikit-Learn Workflow

import numpy as np
from [Link] import load_iris
from sklearn.model_selection import train_test_split
from [Link] import StandardScaler
from [Link] import accuracy_score, classification_report
 # Step 1: Load data
data = load_iris()
X, y = [Link], [Link]
print(f"Features: {data.feature_names}")
print(f"Classes: {data.target_names}")
print(f"Shape: X={[Link]}, y={[Link]}")

# Step 2: Split data

X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.2, random_state=42, stratify=y
)

# Step 3: Preprocess (scale features)

scaler = StandardScaler()
X_train = scaler.fit_transform(X_train) # Fit on training data only
X_test = [Link](X_test) # Apply same scaling to test

# Step 4: Train model (any classifier)

from [Link] import DecisionTreeClassifier
model = DecisionTreeClassifier(random_state=42)
[Link](X_train, y_train)

# Step 5: Evaluate
y_pred = [Link](X_test)
print(f"Accuracy: {accuracy_score(y_test, y_pred):.2f}")
print(classification_report(y_test, y_pred,
target_names=data.target_names))

Cross-Validation
from sklearn.model_selection import cross_val_score, StratifiedKFold

# 5-fold stratified cross-validation

cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
scores = cross_val_score(model, X, y, cv=cv, scoring='accuracy')
print(f"CV Accuracy: {[Link]():.2f} ± {[Link]():.2f}")

Hyperparameter Tuning with GridSearchCV

from sklearn.model_selection import GridSearchCV

param_grid = {
'max_depth': [3, 5, 7, None],
'min_samples_split': [2, 5, 10],
'criterion': ['gini', 'entropy']
}

grid_search = GridSearchCV(
DecisionTreeClassifier(random_state=42),
param_grid,
cv=5,
scoring='accuracy',
n_jobs=-1
)
grid_search.fit(X_train, y_train)
print(f"Best parameters: {grid_search.best_params_}")
print(f"Best CV score: {grid_search.best_score_:.3f}")
4.2 Machine Learning with K-Nearest Neighbours (KNN)
K-Nearest Neighbours (KNN) is one of the simplest and most intuitive machine learning
algorithms. It is a non-parametric, instance-based learning algorithm — it makes no
assumptions about the underlying data distribution.

Core Principle
To classify a new data point, KNN finds the K closest training examples (neighbours) in the
feature space and assigns the majority class among those neighbours. For regression, it
averages their values.

How KNN Works — Step by Step

• Step 1: Store all training data (no explicit training phase)
• Step 2: For a new query point, calculate the distance to all training points
• Step 3: Sort the distances and select the K nearest neighbours
• Step 4: For classification — majority vote among K neighbours
• Step 5: For regression — average the target values of K neighbours

Distance Metrics
• Euclidean Distance (most common): √(Σ(xᵢ - yᵢ)²) — straight-line distance
• Manhattan Distance: Σ|xᵢ - yᵢ| — sum of absolute differences
• Minkowski Distance: (Σ|xᵢ - yᵢ|^p)^(1/p) — generalization (p=2: Euclidean, p=1:
Manhattan)
• Cosine Similarity: used for text and high-dimensional data
• Hamming Distance: used for categorical features

Choosing K
• Small K (e.g., K=1): High variance, low bias, sensitive to noise, overfitting
• Large K: Low variance, high bias, smoother boundaries, underfitting
• Rule of thumb: Start with K = √n (n = number of training samples)
• Best practice: Use cross-validation to find optimal K
• Always try odd K for binary classification to avoid ties
💡 Effect of K on Decision Boundaries
K=1: Very jagged boundary — memorizes training data
K=5: Smoother boundary — good balance
K=N (all data): Always predicts the majority class

KNN Classification — Full Implementation

import numpy as np
import [Link] as plt
from [Link] import load_iris
from sklearn.model_selection import train_test_split
from [Link] import StandardScaler
from [Link] import KNeighborsClassifier
from [Link] import (accuracy_score, confusion_matrix,
classification_report, ConfusionMatrixDisplay)

# Load the Iris dataset

iris = load_iris()
X, y = [Link], [Link]
print(f"Dataset: {[Link][0]} samples, {[Link][1]} features")
print(f"Features: {iris.feature_names}")
print(f"Classes: {iris.target_names}")

# Split: 80% train, 20% test

X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.2, random_state=42, stratify=y
)

# Feature scaling (CRITICAL for KNN — distances must be comparable)

scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = [Link](X_test)

# Train KNN classifier

knn = KNeighborsClassifier(
n_neighbors=5, # K value
weights='uniform', # 'uniform' or 'distance'
metric='euclidean', # Distance metric
algorithm='auto' # 'ball_tree', 'kd_tree', 'brute', 'auto'
)
[Link](X_train_scaled, y_train)

# Predictions
y_pred = [Link](X_test_scaled)
y_prob = knn.predict_proba(X_test_scaled) # Probabilities

# Evaluation
print(f"\nAccuracy: {accuracy_score(y_test, y_pred) * 100:.2f}%")
print("\nClassification Report:")
print(classification_report(y_test, y_pred,
target_names=iris.target_names))

# Confusion Matrix
cm = confusion_matrix(y_test, y_pred)
print("\nConfusion Matrix:")
print(cm)

# Predict a new sample

new_sample = [[5.1, 3.5, 1.4, 0.2]]
new_sample_scaled = [Link](new_sample)
prediction = [Link](new_sample_scaled)
print(f"\nPrediction for {new_sample[0]}:
{iris.target_names[prediction[0]]}")

# Find optimal K using cross-validation

from sklearn.model_selection import cross_val_score
k_values = range(1, 31)
cv_scores = []
for k in k_values:
knn_k = KNeighborsClassifier(n_neighbors=k)
scores = cross_val_score(knn_k, X_train_scaled, y_train, cv=5)
cv_scores.append([Link]())
 best_k = k_values[[Link](cv_scores)]
print(f"\nBest K: {best_k} with CV accuracy: {max(cv_scores):.3f}")

KNN Regression
from [Link] import KNeighborsRegressor
from [Link] import fetch_california_housing
from [Link] import mean_squared_error, r2_score

# Load regression dataset

housing = fetch_california_housing()
X, y = [Link], [Link]

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2,

random_state=42)
scaler = StandardScaler()
X_train_s = scaler.fit_transform(X_train)
X_test_s = [Link](X_test)

knn_reg = KNeighborsRegressor(n_neighbors=5, weights='distance')

knn_reg.fit(X_train_s, y_train)
y_pred = knn_reg.predict(X_test_s)

print(f"MSE: {mean_squared_error(y_test, y_pred):.4f}")

print(f"R² Score: {r2_score(y_test, y_pred):.4f}")

KNN Advantages and Disadvantages

Advantages:
• Simple to understand and implement
• No training phase (lazy learner)
• Naturally handles multi-class problems
• Non-parametric — no assumptions about data distribution
• Adapts to new training data easily
Disadvantages:
• Slow prediction: O(n×d) per query where n=samples, d=features
• High memory: must store all training data
• Sensitive to irrelevant features and feature scale
• Curse of dimensionality: performance degrades with high dimensions
• Requires feature scaling

4.3 Machine Learning with Naïve Bayes Algorithm

Theoretical Foundation
Naïve Bayes is a probabilistic classification algorithm based on Bayes' Theorem with the
'naïve' assumption of conditional independence between features given the class label.

Bayes' Theorem
The theorem describes how to update probabilities based on evidence:
P(Class | Features) = P(Features | Class) × P(Class)
─────────────────────────────────
P(Features)

Where:
P(Class | Features) = Posterior probability (what we want)
P(Features | Class) = Likelihood (probability of features given class)
P(Class) = Prior probability (base rate of class)
P(Features) = Evidence (normalizing constant)

Naïve Assumption (conditional independence):

P(f1, f2, ..., fn | Class) = P(f1|Class) × P(f2|Class) × ... × P(fn|Class)

Decision Rule:
Predicted Class = argmax_c [P(Class=c) × ∏ P(fᵢ | Class=c)]

Why 'Naïve'?
The algorithm assumes features are independent given the class. For example, in spam
detection, it assumes the word 'free' and the word 'win' are independent given that an email
is spam. This is rarely true in reality, but the algorithm works surprisingly well despite this
simplification.

Types of Naïve Bayes Classifiers

Type Features Assumption Use Case
Gaussian NB Continuous features Assumes features follow Medical diagnosis, any
a Gaussian (normal) real-valued features
distribution within each
class.
Multinomial NB Discrete/count features Models feature Text classification,
frequencies (word document categorization
counts). Uses
multinomial distribution.
Bernoulli NB Binary features (0/1) Features are binary — Spam detection,
word present/absent sentiment analysis
(not frequency). Uses
Bernoulli distribution.
Complement NB Discrete features, Improved MultinomialNB Text classification with
imbalanced data for imbalanced datasets. imbalanced classes
Uses complement of each
class.
Categorical NB Categorical features Handles features with Tabular data with
discrete categories. categorical columns

Gaussian Naïve Bayes — Implementation

import numpy as np
from [Link] import load_iris, load_breast_cancer
from sklearn.model_selection import train_test_split, cross_val_score
from sklearn.naive_bayes import GaussianNB, MultinomialNB, BernoulliNB
from [Link] import StandardScaler
from [Link] import (accuracy_score, confusion_matrix,
classification_report, roc_auc_score)
 # ── Gaussian NB on Iris Dataset ──────────────────────────────────
iris = load_iris()
X, y = [Link], [Link]

X_train, X_test, y_train, y_test = train_test_split(

X, y, test_size=0.2, random_state=42, stratify=y
)

# GaussianNB does NOT require feature scaling (it estimates its own
parameters)
gnb = GaussianNB(var_smoothing=1e-9) # var_smoothing adds small value to
avoid zero variance
[Link](X_train, y_train)

y_pred = [Link](X_test)
y_prob = gnb.predict_proba(X_test)

print("=== Gaussian Naive Bayes - Iris Dataset ===")

print(f"Accuracy: {accuracy_score(y_test, y_pred) * 100:.2f}%")
print(f"\nClassification Report:")
print(classification_report(y_test, y_pred,
target_names=iris.target_names))

# Inspect learned parameters

print(f"Class priors: {gnb.class_prior_}") # P(class)
print(f"Class means (theta):\n{gnb.theta_}") # Mean of each feature
per class
print(f"Class variances:\n{gnb.var_}") # Variance of each
feature per class

# Cross-validation
scores = cross_val_score(gnb, X, y, cv=10, scoring='accuracy')
print(f"10-Fold CV Accuracy: {[Link]():.3f} ± {[Link]():.3f}")

# Predict new sample

new_sample = [[6.1, 2.8, 4.7, 1.2]]
pred = [Link](new_sample)
prob = gnb.predict_proba(new_sample)
print(f"\nNew sample prediction: {iris.target_names[pred[0]]}")
print(f"Probabilities: {dict(zip(iris.target_names, prob[0].round(3)))}")

Multinomial NB for Text Classification

from sklearn.naive_bayes import MultinomialNB
from sklearn.feature_extraction.text import CountVectorizer,
TfidfVectorizer
from [Link] import Pipeline
from sklearn.model_selection import train_test_split
from [Link] import accuracy_score, classification_report

# Sample email dataset (simplified spam detection)

emails = [
"Win a free iPhone now click here", "Get free money now limited offer",
"Congratulations you won a prize claim now", "Free cash reward win
today",
"Click here for free gift limited time", "Buy cheap medications online
now",
"Meeting scheduled for tomorrow at 9am", "Please review the attached
report",
 "Can we discuss the project deadline", "Team lunch this Friday at
noon",
"Your invoice is attached for review", "Quarterly performance review
attached",
"Happy birthday! Hope you have a great day", "See you at the conference
next week",
]
labels = [1,1,1,1,1,1, 0,0,0,0,0,0, 0,0] # 1=spam, 0=ham

# Split data
X_train, X_test, y_train, y_test = train_test_split(
emails, labels, test_size=0.3, random_state=42
)

# Build pipeline: TF-IDF vectorization + Multinomial NB

pipeline = Pipeline([
('tfidf', TfidfVectorizer(
stop_words='english',
ngram_range=(1, 2), # Unigrams and bigrams
max_features=1000
)),
('classifier', MultinomialNB(alpha=1.0)) # alpha = Laplace smoothing
])

[Link](X_train, y_train)
y_pred = [Link](X_test)

print("=== Multinomial NB - Spam Classifier ===")

print(f"Accuracy: {accuracy_score(y_test, y_pred) * 100:.2f}%")
print(classification_report(y_test, y_pred, target_names=['Ham', 'Spam']))

# Test with new messages

test_messages = [
"Free money win now click here",
"Project meeting tomorrow 10am",
"You have won exclusive prize",
"Please send me the report"
]
predictions = [Link](test_messages)
probs = pipeline.predict_proba(test_messages)
for msg, pred, prob in zip(test_messages, predictions, probs):
label = "SPAM" if pred == 1 else "HAM "
print(f"[{label}] ({prob[1]*100:.1f}% spam) {msg}")

Bernoulli Naïve Bayes

from sklearn.naive_bayes import BernoulliNB
from sklearn.feature_extraction.text import CountVectorizer

# Bernoulli NB — uses binary presence/absence of words (not counts)

vectorizer = CountVectorizer(binary=True) # binary=True converts counts to
0/1
X_bin = vectorizer.fit_transform(emails)

X_train, X_test, y_train, y_test = train_test_split(

X_bin, labels, test_size=0.3, random_state=42
)

bnb = BernoulliNB(alpha=1.0, binarize=None) # binarize=None since already

 binary
[Link](X_train, y_train)
y_pred = [Link](X_test)
print(f"Bernoulli NB Accuracy: {accuracy_score(y_test, y_pred) *
100:.2f}%")

Comparing All Naïve Bayes Variants

from [Link] import load_breast_cancer
from sklearn.naive_bayes import GaussianNB, BernoulliNB
from sklearn.model_selection import StratifiedKFold, cross_val_score

cancer = load_breast_cancer()
X, y = [Link], [Link]

cv = StratifiedKFold(n_splits=10, shuffle=True, random_state=42)

classifiers = {
'Gaussian NB': GaussianNB(),
'Bernoulli NB': BernoulliNB(),
}

print("=== Breast Cancer Classification Comparison ===")

for name, clf in [Link]():
scores = cross_val_score(clf, X, y, cv=cv, scoring='accuracy')
print(f"{name}: {[Link]()*100:.2f}% ± {[Link]()*100:.2f}%")

Laplace Smoothing
A critical concept in Naïve Bayes: if a feature value never appears in the training data for a
class, its probability is 0 — causing the entire probability to be 0 (zero-frequency problem).
Laplace smoothing (add-one smoothing) adds a small count α to all feature counts to prevent
this.
# Without smoothing: P(word|class) = 0 if word not seen → kills all
probabilities

# With Laplace smoothing (alpha=1):

# P(word|class) = (count(word,class) + alpha) / (total_words_in_class +
alpha × vocab_size)

MultinomialNB(alpha=1.0) # Default: Laplace smoothing (alpha=1)

MultinomialNB(alpha=0.1) # Lidstone smoothing (smaller alpha)
MultinomialNB(alpha=0) # No smoothing (dangerous!)

Naïve Bayes Advantages and Disadvantages

Advantages:
• Extremely fast — training O(n×d) and prediction O(c×d) where c=classes
• Works well with small datasets
• Handles high-dimensional data (especially text)
• Robust to irrelevant features
• Requires little data to estimate parameters
 • Naturally handles multi-class classification
• Produces probability estimates, not just labels
• No hyperparameter tuning required (except smoothing)
Disadvantages:
• Strong independence assumption is often violated
• Cannot capture feature interactions
• Probability estimates can be poorly calibrated
• Continuous features require distribution assumption (Gaussian NB)
• Zero-frequency problem requires smoothing

When to Use Each Algorithm

Algorithm Best Used When Key Characteristics
K-Nearest Neighbours • Small to medium datasets (slow • Feature scaling required
at scale) • No assumptions about data
• Non-linear boundaries distribution
• Low-dimensional data • Works for both classification
• When interpretability matters and regression
Naïve Bayes • Text classification (spam, • Very fast training and
sentiment) prediction
• Real-time prediction needed • Independence assumption
• Very small datasets • Works best with
• High-dimensional sparse data discrete/categorical features

Model Evaluation Metrics Summary

from [Link] import (
accuracy_score, # Overall correct predictions
precision_score, # Of predicted positives, how many correct?
recall_score, # Of actual positives, how many found?
f1_score, # Harmonic mean of precision and recall
confusion_matrix, # TP, FP, FN, TN breakdown
roc_auc_score, # Area under ROC curve
mean_squared_error, # For regression: average squared error
r2_score # For regression: variance explained
)

# Accuracy = (TP + TN) / Total

# Precision = TP / (TP + FP) — minimizes false positives
# Recall = TP / (TP + FN) — minimizes false negatives
# F1-Score = 2 × (Precision × Recall) / (Precision + Recall)

💡 Key Takeaway
The workflow for any ML problem:
1. Collect and explore data
2. Preprocess (handle missing values, encode, scale)
3. Split into train/test sets
4. Train multiple algorithms
5. Evaluate using cross-validation and metrics
6. Tune hyperparameters (GridSearchCV)
7. Select best model and deploy
— End of AI Study Notes —