Module 2

Data and Knowledge

Management
 Managing Data
● All IT applications depend on high-quality data that are accurate,
complete, timely, consistent, accessible, relevant, and concise.
However, managing data is challenging because:
1. Data growth is exponential – Organizations must store historical
data while rapidly adding new data. Example: Walmart manages
petabytes of customer and sales data.
2. Data are scattered – Collected by different people, devices, and
methods, stored in multiple servers, databases, formats, and
languages, making integration complex.
 Managing Data
3. Multiple Sources of Data
● Internal: Company databases and documents.

● Personal: Employee opinions or experiences.

● External: Government reports, commercial databases, corporate

websites.
● Web/Clickstream: Tracks user clicks and browsing behavior.

● Example: Amazon uses clickstream data to recommend products.

 Managing Data
4. Emerging Data Sources:
● New inputs like blogs, podcasts, videocasts, RFID tags, and wireless
sensors keep expanding the data pool.

5. Data Degradation & Rot:

● Data changes over time – e.g., customers move, companies merge.

● Storage media decay or become outdated.

● Example: 8-track tapes are useless today without compatible players.

 Managing Data
6. Data Security & Legal Issues:
● Data quality, integrity, and protection are easily compromised.

● Legal requirements vary across countries and industries.

7. Data Duplication & Inconsistency:

● Different business systems (CRM, billing, marketing) store the same
data differently, leading to conflicts.
● Example: Customer addresses in marketing and billing systems may
differ, preventing a unified customer view.
 Database Approach
● A data file is a collection of logically related records.
● In a file management environment, each application has a specific
data file related to it.
● This file contains all of the data records the application requires.

In traditional file systems, each application has its own separate

data file, causing:
1. Data Redundancy – Duplicate data in multiple files.
2. Data Isolation – Data in one file is not easily accessible to other
applications.
3. Data Inconsistency – Different files may have conflicting data.
 Database Approach Contd..
Database Approach:
● Uses a Database Management System (DBMS) to provide
centralized access to all data.
● Reduces redundancy, ensures consistency, and integrates data across
applications.

Example:
● A university central database stores student records.

● Instead of clubs, sports departments, and professors keeping separate

files, all can access the same updated record, ensuring your name and
address are consistent everywhere.
 Database Approach Contd..
Database systems provide three key benefits:
1. Data Security – Centralized storage is protected with strong security
measures to prevent loss or unauthorized access. Example: A bank
database protects all customer accounts with strict security protocols.
2. Data Integrity – Data follow defined rules and constraints, ensuring
accuracy. Example: A Social Security Number field accepts only
numeric values, preventing errors.
3. Data Independence – Applications are separate from data, allowing
multiple applications to access the same data without duplication.
Example: A university database can be accessed by admissions,
finance, and exam systems simultaneously.
 Database Approach Contd..

Figure 2.1: Database management system (University Database)

 Data Hierarchy
Data are organized in a hierarchy that begins with bits and proceeds
all the way to databases.
1. Bit: A bit (binary digit) represents the smallest unit of data a computer
can process. The term binary means that a bit can consist only of a 0 or
a 1.
2. Byte: A group of eight bits, called a byte, represents a single character.
A byte can be a letter, a number, or a symbol.
3. Field: Logical group of characters; represents a single data item.
Example: Student Name or Social Security Number. Can also store
images or audio (e.g., driver’s photo in a license database).
 Data Hierarchy Contd..
4. Record: Collection of related fields. Example: A student’s name,
courses, and grades.
5. File/Table – Collection of related records. Example: All student grades
for a course.
6. Database – Collection of related files/tables. Example: University
student database containing personal, academic, and financial records.
 Data Hierarchy Contd..

Figure 2.2: Hierarchy of data for a computer-based file.

 Data-Warehousing Defination
● Data Warehouse stores huge amount of data, which is typically
collected from multiple heterogeneous sources like files, DBMS, etc.

● The goal is to produce statistical results that may help in decision

making.

● The term Data Warehouse was defined by Bill Inmon in 1990, in the
following way:
A Data Warehouse is a subject-oriented, integrated, non-volatile,
and time variant collection of data in support of management’s decision
making process.
 Data-Warehousing contd..
● A Data Warehouse is a group of data specific to the entire
organization, not only to a particular group of users.

● It is not used for daily operations and transaction processing but used
for making decisions.
 Data-Warehousing Contd..
Subject-Oriented Data:
Data that gives information about a particular subject instead of about a
company’s operations.
 Data-Warehousing Contd..
Integrated Data:
A data warehouse integrates various heterogeneous data sources like
RDBMS, flat files, and online transaction records.
● It requires performing data cleaning and integration during data warehousing to
ensure consistency in naming conventions, attributes types, etc., among different
data sources.
●
 Data-Warehousing contd..
Time Variant Data:
● Historical data is kept in a data warehouse. For example, one can retrieve data
from 3 months, 6 months, 12 months, or even older data from a data warehouse.
● This contrasts with a transactions system, where often only the most recent data
is kept.

Non-volatile:
● Once data is in the data warehouse, it will not change.

● So, historical data in a data warehouse should never be altered.

Data Warehouse Architecture
 Data Warehouses adopt a three-tier architecture:

❖ Bottom Tier:
● It is a warehouse database server, that is generally a RDBMS.
● Using Application Programming interfaces (called as gateways), data is
extracted from operational and external sources.
● Gateways like, ODBC(Open Database Connection), OLE-DB (Open linking
and embedding for database), JDBC (Java Database Connection) is supported
by underlying DBMS.
● The tier also contains metadata repository, which stores information about the
data warehouse and its contents.
● Metadata are data about data.
● Metadata are the data that define warehouse objects.
 ❖ Middle Tier:
● The middle tier is an OLAP server that is typically implemented using either
1) a relational OLAP (ROLAP) model (i.e., an extended relational DBMS that
maps operations on multidimensional data to standard relational operations); or
2) a multi-dimensional OLAP (MOLAP) model (i.e., a special-purpose server that
directly implements multidimensional data and operations).
● OLAP servers present business users with multidimensional data from data
warehouses or data marts, without concerns regarding how or where the data are
stored.

❖ Top Tier:
● The top tier is a front-end client layer, which contains query and reporting tools,
analysis tools, and/or data mining tools (e.g., trend analysis, prediction, and so
on).
 Data Warehouse and Data Marts
❖ Data Mart:
● A data mart contains a subset of corporate-wide data that is of value to a specific
group of users.
● The scope is confined to specific selected subjects.
● For example, a marketing data mart may confine its subjects to customer, item,
and sales.
● The data contained in data marts tend to be summarized.
● Data marts are usually implemented on low-cost departmental servers that are
Unix/Linux or Windows based.
● The implementation cycle of a data mart is more likely to be measured in weeks
rather than months or years.
● However, it may involve complex integration in the long run if its design and
planning were not enterprise-wide.
Data Mart Contd..
Depending on the source of data, data marts can be categorized as
independent or dependent.
● Independent data marts are sourced from data captured from one or more
operational systems or external information providers, or from data generated
locally within a particular department or geographic area.
● Dependent data marts are sourced directly from enterprise data warehouses.
 Data Warehouse Vs Data Mart
Sr. No. Data Warehouse Data Mart
1. Data Warehouse is a centralized system. Data Mart is a decentralized system.
2. The designing process of Data Warehouse The designing process of Data Mart is easy.
is quite difficult.

3. Data warehouse is top-down model. Data Mart is a bottom-up model.

4. The size of the Data Warehouse may range The Size of Data Mart is less than 100 GB.
from 100 GB to 1 TB+.

5. Implementation takes months to years. Implementation is done usually in months.

6. It consists of multiple subjects. It consist of a single subject of concern to
the user.
 Big Data
● Big Data refers to extremely large, complex, and rapidly growing
datasets that are difficult to store, process, or analyze using traditional
data management tools.
It is often described by the 5 Vs:
● Volume – Large amounts of data (terabytes, petabytes, etc.).
● Velocity – Data is generated and processed at high speed (real-time or
near real-time).
● Variety – Data comes in different formats: structured (tables), semi-
structured (JSON, XML), and unstructured (text, images, videos).
● Veracity – Data can be uncertain, incomplete, or inconsistent.
● Value – The usefulness of data in decision-making.
 Big Data
Example: Imagine YouTube. Every minute:
● Users upload 500+ hours of video (Volume).

● New videos are available to watch instantly (Velocity).

● Videos, comments, likes, and user analytics all have different formats
(Variety).
● Some videos have misleading titles or poor-quality data (Veracity).

● YouTube uses this data to recommend videos and run targeted ads
(Value).
 Knowledge Management
● Knowledge management (KM) is a process that helps organizations
manipulate important knowledge that comprises part of the organization’s
memory, usually in an unstructured format.
● Knowledge Management (KM) is the process of capturing, organizing,
sharing, and effectively using an organization’s knowledge and experience to
improve decision-making, efficiency, and innovation.
● It focuses on ensuring that valuable information — whether it’s in
documents, databases, or employees’ minds — is accessible to the right
people at the right time.
 Knowledge Management
Key Steps in Knowledge Management:
● Creation – Developing new knowledge from research, experience, or
innovation.
● Storage – Organizing and saving knowledge in repositories (databases,
manuals, intranets).
● Sharing – Distributing knowledge across teams or the organization.
● Application – Using knowledge to solve problems, make decisions, or
improve processes.
 Knowledge Management
Example: A hospital implements a KM system where
● Creation – Doctors record new treatment techniques after research and
patient cases.
● Storage – These treatments are stored in a central medical knowledge
database.
● Sharing – The database is accessible to all doctors in the hospital through
the intranet.
● Application – When a similar patient case occurs, doctors quickly find and
apply proven treatments, improving recovery rates.
 Knowledge Management
❖ Knowledge:
● Data are a collection of facts, measurements, and statistics;
● Information is organized or processed data that are timely and accurate.
● Knowledge is information that is contextual, relevant, and useful.
Knowledge is information in action. Also called intellectual capital/assets.

Example:
● Data – List of daily temperatures recorded in your city for a week.

● Information – A weather report showing the weekly temperature trend.

● Knowledge – Deciding to carry an umbrella or wear light clothes based on

the forecast and your outdoor plans.
 Knowledge Management
❖ Types of Knowledge:
● Explicit Knowledge – Easily documented, stored, and shared.
○ Example: User manuals, standard operating procedures.
○ A math textbook explaining formulas and problem-solving steps.
● Tacit Knowledge – Personal, experience-based, hard to formalize.
○ Example: A chef’s unique cooking skills learned over years of practice.
○ A skilled carpenter’s “feel” for choosing the right type of wood just by
touching and smelling it.
 Knowledge Management System
• Knowledge management systems (KMSs) refer to the use of modern
information technologies—the Internet, intranets, extranets, databases—to
systematize, enhance, and expedite intrafirm and interfirm knowledge
management.
• KMSs are intended to help an organization cope with turnover, rapid change,
and downsizing by making the expertise of the organization’s human capital
widely accessible.
The KMS Cycle
 KMS Cycle
The cycle works as follows:
1. Create knowledge: Knowledge is created as people deter mine new ways
of doing things or develop know-how. Sometimes external knowledge is
brought in.
2. Capture knowledge: New knowledge must be identified as valuable and
be represented in a reasonable way.
3. Refine knowledge: New knowledge must be placed in context so that it is
actionable. This is where tacit qualities (human insights) must be captured
along with explicit facts.
4. Store knowledge: Useful knowledge must then be stored in a reasonable
format in a knowledge repository so that others in the organization can
access it.
 KMS Cycle
5. Manage knowledge: Like a library, the knowledge must be kept current. It
must be reviewed regularly to verify that it is relevant and accurate.
6. Disseminate knowledge: Knowledge must be made available in a useful
format to anyone in the organization who needs it, anywhere and anytime
 Business Intelligence
● Business Intelligence (BI) is the process of collecting, integrating,
analyzing, and presenting business data to help organizations make better
decisions.
● It uses tools, technologies, and processes to turn raw data into meaningful
insights — usually in the form of dashboards, reports, charts, or visual
analytics.
 Business Intelligence
Example: A retail chain uses BI software to analyze its sales data from all
stores.
● Data collected: Daily sales, customer demographics, inventory levels,
and regional trends.
● BI Insight: The dashboard shows that winter jackets sell 60% more in
northern cities during October–December.
● Business Decision: The company increases jacket stock in northern
stores before winter and offers targeted promotions — boosting revenue.
 Managers and Decision Making
● Management uses resources (people, money, materials, information) as
inputs to achieve organizational goals (outputs), aiming for high
productivity.
Managers perform three main roles:
1. Interpersonal roles: figurehead, leader, liaison
2. Informational roles: monitor, disseminator, spokesperson, analyzer
3. Decisional roles: entrepreneur, disturbance handler, resource allocator,
negotiator
 Managers and Decision Making
❖ Decision Making Process:
● Intelligence phase – identify and define problems/opportunities.
● Design phase – build and test models, set evaluation criteria.
● Choice phase – select and implement the best solution.
If the solution fails, managers revisit earlier phases.
Decision Making Process
 Decision Making Process
The diagram represents the Decision-Making Process in three
main phases: Intelligence, Design, and Choice, along with validation,
testing, and feedback loops to ensure success.

1. Reality (Problem Identification):

● The process starts with examining reality to identify an existing
problem or need.
● This leads into the Intelligence Phase.
 Decision Making Process
2. Intelligence Phase – “What is the problem?”
● In this stage, the problem is studied in detail.

● Data is collected, situations are analyzed, and the root cause is

identified.
● Example: In agriculture, detecting that crops are showing signs of
disease.

3. Design Phase – “What are my options?”

● Possible solutions are brainstormed and designed.

● Models or approaches are developed to tackle the identified problem.

● Example: Using AI models, pesticides, or new irrigation methods to

solve crop health issues.
 Decision Making Process
4. Choice Phase – “Pick an option and decide how to implement
it.”
● A suitable option is selected from the available solutions.

● An implementation plan is created, specifying how the chosen

solution will be executed.

5. Implementation of Solution
● The chosen solution is put into action in the real world.

● Example: Deploying a YOLOv8 model in the field to detect crop

diseases in real-time.
 Decision Making Process
4. Choice Phase – “Pick an option and decide how to implement
it.”
● A suitable option is selected from the available solutions.

● An implementation plan is created, specifying how the chosen

solution will be executed.

5. Implementation of Solution
● The chosen solution is put into action in the real world.

● Example: Deploying a YOLOv8 model in the field to detect crop

diseases in real-time.
 Decision Making Process
6. Verification & Testing of Proposed Solution
● After implementation, the solution is tested to verify if it works as
expected.
● Question: “Did this option work as proposed?”
○ Yes → SUCCESS: Problem solved, happy outcome.
○ No → FAILURE: Solution didn’t work as expected.

7. Feedback Loop
● If Failure: The process loops back to earlier stages (Design or Choice
Phase) to rework the solution.
● If Success: The solution becomes validated and can be applied in similar
scenarios in the future.
Business Intelligence Applications for Data Analysis
A variety of BI applications for analyzing data are available. They
include:
1. Multidimensional analysis (also called online analytical processing, or
OLAP),
2. Data mining, and
3. Decision support systems
 Business Intelligence Applications for Presenting Results
● Dashboards
● Data Visualization Technologies