Module 2- Data and

knowledge management
Introduction
● Data
● Forms of Data
● Data and Information
● Fundamental concepts of data
Data types
Field
Record
File
Databases
Terminologies associated with database
Managing Data

All IT applications require data. These data should be of high quality ,

meaning that they should be accurate, complete, timely, consistent,
accessible, relevant, and concise. Unfortunately, the process of acquiring,
keeping, and managing data is becoming increa
Managing data
● Data Governace -policies, ethics,standards,Guidelines
● Master data-ERP
● Tranction data
Database Management System
DBMS Building blocks

Modeling language

Data structure

Database querry language

Transaction mechanism
Functions of DBMS
● Creation, Modification, deletion of tables.
● Manipulation of records in tables by sorting/filtering them.
● Serching of dsired records and updating/deleting them
● Mathematical/logical operations on values in the fields
● Design of User interface to linkand automate various modules of application
● Processing of user queries and presentation of results in tabular format.
● Linking with other databases.
Types of databases
Depending upon the usage requirements, there are following types of databases available in the market

● Centralised database.
● Distributed database.
● Personal database.
● End-user database.
● Commercial database.
● Relational database.
● Cloud database.
Centralised Database

The information(data) is stored at a centralized location and the users from different locations can access this data. This
type of database contains application procedures that help the users to access the data even from a remote location.

Various kinds of authentication procedures are applied for the verification and validation of end users, likewise, a
registration number is provided by the application procedures which keeps a track and record of data usage. The local
area office handles this thing.
Distributed Database

Just opposite of the centralized database concept, the distributed database has contributions from the common database
as well as the information captured by local computers also. The data is not at one place and is distributed at various sites
of an organization. These sites are connected to each other with the help of communication links which helps them to
access the distributed data easily.

You can imagine a distributed database as a one in which various portions of a database are stored in multiple different
locations(physical) along with the application procedures which are replicated and distributed among various points in a
network.
Personal Database
Data is collected and stored on personal computers which is small and easily manageable. The data is generally used by
the same department of an organization and is accessed by a small group of people.

End User Database

The end user is usually not concerned about the transaction or operations done at various levels and is only aware of the
product which may be a software or an application. Therefore, this is a shared database which is specifically designed for
the end user, just like different levels’ managers. Summary of whole information is collected in this database.

Commercial Database

These are the paid versions of the huge databases designed uniquely for the users who want to access the information
for help. These databases are subject specific, and one cannot afford to maintain such a huge information. Access to such
databases is provided through commercial links.
Relational Databases

Relational Databases
These databases are categorized by a set of tables where data gets fit into a pre-defined category. The table consists of
rows and columns where the column has an entry for data for a specific category and rows contains instance for that data
defined according to the category. The Structured Query Language (SQL) is the standard user and application program
interface for a relational database.

There are various simple operations that can be applied over the table which makes these databases easier to extend,
join two databases with a common relation and modify all existing applications.
Cloud Databases

Now a day, data has been specifically getting stored over clouds also known as a virtual environment, either in a hybrid
cloud, public or private cloud. A cloud database is a database that has been optimized or built for such a virtualized
environment. There are various benefits of a cloud database, some of which are the ability to pay for storage capacity and
bandwidth on a per-user basis, and they provide scalability on demand, along with high availability.

A cloud database also gives enterprises the opportunity to support business applications in a software-as-a-service
deployment.
Data Warehousing
A Data Warehouse (DW) is a relational database that is designed for query and analysis rather than transaction
processing. It includes historical data derived from transaction data from single and multiple sources.

A Data Warehouse provides integrated, enterprise-wide, historical data and focuses on providing support for
decision-makers for data modeling and analysis.

A Data Warehouse is a group of data specific to the entire organization, not only to a particular group of users.

It is not used for daily operations and transaction processing but used for making decisions.

A Data Warehouse can be viewed as a data system with the following attributes:

● It is a database designed for investigative tasks, using data from various applications.
● It supports a relatively small number of clients with relatively long interactions.
● It includes current and historical data to provide a historical perspective of information.
● Its usage is read-intensive.
● It contains a few large tables.

"Data Warehouse is a subject-oriented, integrated, and time-variant store of information in support of management's
decisions."
Data Marts
A Data Mart is a subset of a directorial information store, generally oriented to a specific purpose or primary
data subject which may be distributed to provide business needs. Data Marts are analytical record stores
designed to focus on particular business functions for a specific community within an organization. Data marts
are derived from subsets of data in a data warehouse, though in the bottom-up data warehouse design
methodology, the data warehouse is created from the union of organizational data marts.

Reasons for creating a data mart

● Creates collective data by a group of users
● Easy access to frequently needed data
● Ease of creation
● Improves end-user response time
● Lower cost than implementing a complete data warehouses
● Potential clients are more clearly defined than in a comprehensive data warehouse
● It contains only essential business data and is less cluttered.
 BIG DATA
ssage exchanges, putting comments etc.
What Happens online in 60 seconds?
Big data
Big data is a collection of data that is larger, more complex than traditional data, and yet growing
exponentially with time. It is so huge that no traditional data management software or tool can manage,
store, or can process it efficiently. So, it needs to be processed step by step via different methodologies.

Applications of Big Data are

● Banking and Securities
● Communications, Media and Entertainment
● Healthcare Providers
● Education
● Manufacturing and Natural Resources
● Government
● Insurance
● Retail and Wholesale trade
Types Of Big Data
Following are the types of Big Data:

1. Structured Data:
This type of data is highly organized and formatted in a predefined manner, making it easily searchable and analyzable
using traditional database management systems. Examples include data in relational databases, spreadsheets, and
data warehouses, often containing numerical values, dates, and text in clearly defined fields.
2. Unstructured Data:
Unstructured data lacks a predefined structure or format, making it challenging to store and analyze using conventional
methods. It comprises a vast majority of the world's data and includes various forms such as text documents, images,
audio files, video files, social media posts, and sensor data.
3. Semi-structured Data:
This type of data possesses some organizational structure but does not conform to the rigid schema of structured data.
It contains tags or markers that provide a degree of organization, allowing for easier parsing and analysis compared to
unstructured data. Examples include XML files, JSON documents, log files, and email data.
Characteristics Of Big Data

Big data can be described by the following characteristics:

● Volume
● Variety
● Velocity
● Veracity
● value
1. Volume – Scale of Data
The name ‘Big Data’ itself is related to a size which is enormous. Volume is a huge amount of data. To determine the
value of data, size of data plays a very crucial role. If the volume of data is very large, then it is actually considered as
a ‘Big Data’. This means whether a particular data can actually be considered as a Big Data or not, is dependent upon
the volume of data. Hence while dealing with Big Data it is necessary to consider a characteristic ‘Volume’.
Example: In the year 2016, the estimated global mobile traffic was 6.2 Exabytes (6.2 billion GB) per month.
Also, by the year 2020 we will have almost 40000 Exabytes of data.

2. Velocity – Speed of Data

Velocity refers to the high speed of accumulation of data. In Big Data velocity data flows in from sources like
machines, networks, social media, mobile phones etc. There is a massive and continuous flow of data. This
determines the potential of data that how fast the data is generated and processed to meet the demands. Sampling
data can help in dealing with the issue like ‘velocity’.
Example: There are more than 3.5 billion searches per day are made on Google. Also, Facebook users are
increasing by 22%(Approx.) year by year.
3. Variety – Diversity of Data
It refers to nature of data that is structured, semi-structured and unstructured data. It also refers to
heterogeneous sources. Variety is basically the arrival of data from new sources that are both inside and
outside of an enterprise. It can be structured, semi-structured and unstructured.

4. Veracity – Trustworthiness of Data

It refers to inconsistencies and uncertainty in data, that is data which is available can sometimes get messy
and quality and accuracy are difficult to control. Big Data is also variable because of the multitude of data
dimensions resulting from multiple disparate data types and sources.
Example: Data in bulk could create confusion whereas less amount of data could convey half or
Incomplete Information.

5. Value – Usefulness of Data

After having the 4 V’s into account there comes one more V which stands for Value. The bulk of Data having
no Value is of no good to the company, unless you turn it into something useful. Data in itself is of no use or
importance but it needs to be converted into something valuable to extract Information. Hence, you can state
that Value is the most important V.
How do we store and process data
Frameworks

● Cassandra
● Hadoop
● Spark
To store and process big data, organizations typically rely on distributed computing frameworks and scalable storage
solutions. One popular framework for processing big data is Apache Hadoop, which includes HDFS (Hadoop Distributed
File System) for storage and MapReduce for processing. Another framework is Apache Spark, which is known for its speed
and ability to perform complex analytics.

For example, a retail company might use Hadoop to store and process transaction data from thousands of stores. The data
could be analyzed to identify trends in customer purchasing behavior, which could then inform marketing strategies.

In the context of cloud computing, services like Tencent Cloud's Cloud Data Lake Storage (CDS) can be used for scalable
storage, and services like Tencent Cloud's Big Data Processing Service (TBPS) can handle the processing. These
cloud-based solutions offer flexibility, scalability, and cost-effectiveness compared to on-premises solutions.

Additionally, real-time processing can be achieved using technologies like Apache Kafka and Apache Flink, which can
handle high volumes of data streams in real-time. For instance, a social media platform might use Kafka to process and
analyze user activity in real-time to provide personalized recommendations.

By leveraging these technologies and services, organizations can effectively manage and derive insights from their big
data.
Knowledge Management
Knowledge management is an activity practised by enterprises all over the world. In the process of knowledge
management, these enterprises comprehensively gather information using many methods and tools.

Then, gathered information is organized, stored, shared, and analyzed using defined techniques.

The analysis of such information will be based on resources, documents, people and their skills.

Properly analyzed information will then be stored as 'knowledge' of the enterprise. This knowledge is later used for
activities such as organizational decision making and training new staff members.
The Knowledge Management Process
Business Intelligence
● BI(Business Intelligence) is a set of processes, architectures, and technologies that convert raw data
into meaningful information that drives profitable business actions. It is a suite of software and services
to transform data into actionable intelligence and knowledge.
● BI has a direct impact on organization’s strategic, tactical and operational business decisions. BI
supports fact-based decision making using historical data rather than assumptions and gut feeling.
● BI tools perform data analysis and create reports, summaries, dashboards, maps, graphs, and charts to
provide users with detailed intelligence about the nature of the business.
How Business Intelligence systems are implemented?

Step 1) Raw Data from corporate databases is extracted. The data could be spread across multiple systems
heterogeneous systems.

Step 2) The data is cleaned and transformed into the data warehouse. The table can be linked, and data
cubes are formed.
Step 3) Using BI system the user can ask quires, request ad-hoc reports or conduct any other analysis.
Functions of BI
● Data Mining: It is the process of analyzing large volumes of data to gather more information. Datasets, statistics, and
machine learning are used for data mining.
● Data Preparation: Data from internal and external sources is collected, cleaned, and formatted to keep it ready for
data analytics.
● Types of Analytics: Business intelligence is capable of providing different types of analytics based on the
organization’s requirements. For example, preliminary data is used for descriptive analytics; descriptive analytics
data is used for statistical analytics; predictive analytics is a combination of statistical analytics, data mining, and
machine learning.
● Querying: An employee asking the BI tool for specification information is called querying. The tool uses computing
power to collect data from datasets and processes it to provide an answer to the query.
● Data Visualization: Visualization is the technique of presenting data in a graphic form in real-time. The reports
generated by the BI tool are presented in charts, graphs, holograms, heat maps, etc., for easy understanding.
Difference between MIS and BI

Scope and Focus

Management Information System or MIS is a part of the business intelligence framework.
Traditionally, enterprises used MIS mainly for revenue and expense reporting. However,
they had to face several issues with inaccuracies, inconsistent data, and other system
complications. While business intelligence deals with the analysis of products and
customers, MIS is used to analyze the business in terms of finances.
Difference between MIS and BI

Difference in Reporting
MIS delivers aggregate data reports which are not enough to get a complete picture of the
business situation. It doesn’t offer in-depth insights that help the managers identify the problem.
For example, the MIS report gives aggregate data about income and expenses. This doesn’t
include transactional data that are key to identifying the root cause of the issue. The managers
have to run multiple queries to gather the relevant information.
Business intelligence tools provide in-depth analytics for the given data. Moreover, the reports
are generated in real-time. The dashboard of a BI tool allows employees to get a macro and
micro view of the same report just with a few clicks. Business intelligence not only works faster
but also requires lesser computing power as it has a higher processing capacity.
● Accurate Insights: Unlike MIS, the reports generated by BI tools don’t have accuracy
issues. Data mining prevents data redundancy and improves big data quality.
● Reporting Automation: BI processes can be automated to complete recurring tasks
without human interference. Once the reporting setup is established, the reports are
automatically generated and shared with the relevant employees.
● Reliability: BI is more reliable than MIS as it uses advanced tools and technologies to
provide real-time analytics. Managers make decisions based on the latest
information instead of outdated data.
● Flexibility & Scalability: The business intelligence framework can be scaled to suit an
enterprise of any size and volume. The processes and elements can be customized
to suit the specific requirements of different businesses.
Difference between MIS and BI

Self-Servicing
Business intelligence tools allow self-servicing within organizations by empowering
employees to access data and run analytics as and when necessary. The dashboards can
be accessed from different platforms such as web browsers and mobile apps. While MIS
is complicated and confusing, BI is elaborate yet efficient and user-friendly. Automation is
possible with business intelligence.
Impact of BI on Decision Making
● Improving interactivity
● Data visualization
● Connection to databases
● Predictive analysis
● Application integration
● Mobile business integration
BI Tools
1. SAP Business Objects

SAP Business Objects is a business intelligence software which offers comprehensive reporting, analysis and
interactive data visualisation. The platform focuses heavily on categories such as Customer Experience (CX)
and CRM, digital supply chain, ERP and more. What’s really nice about this platform is the self-service,
role-based dashboards its offers enabling users to build their own dashboards and applications. SAP is a
robust software intended for all roles (IT, end uses and management) and offers tons of functionalities in one
platform. The complexity of the product, however, does drive up the price so be prepared for that.
2. Datapine

Datapine is an all-in-one business intelligence platform that facilitates the complex process of data analytics
even for non-technical users. Thanks to a comprehensive self-service analytics approach, datapine’s solution
enables data analysts and business users alike to easily integrate different data sources, perform advanced
data analysis, build interactive business dashboards and generate actionable business insights.

3. MicroStrategy

MicroStrategy is an enterprise business intelligence tool that offers powerful (and high speed) dashboarding
and data analytics, cloud solutions and hyperintelligence. With this solution, users can identify trends,
recognise new opportunities, improve productivity and more. Users can also connect to one or various
sources, whether the incoming data is from a spreadsheet, cloud-based or enterprise data software. It can be
accessed from your desktop or via mobile. Setup, however can involve multiple parties and some rather
extensive knowledge of the application in order to get started.
5. Yellowfin BI

Yellowfin BI is a business intelligence tool and ‘end-to-end’ analytics platform that combines visualisation,
machine learning, and collaboration. You can also easily filter through tons of data with intuitive filtering (e.g.
checkboxes and radio buttons) as well open up dashboards just about anywhere (thanks to this tool’s
flexibility in accessibility (mobile, webpage, etc.). The nice thing about this BI tool is that you can easily take
dashboards and visualisations to the next level using a no code/low code development environment.

7. Zoho Analytics

Zoho Analytics is great BI tool for in-depth reporting and data analysis. This business intelligence tool has
automatic data syncing and can be scheduled periodically. You can easily build a connector by using the
integration APIs. Blend and merge data from different sources and create meaningful reports. With an easy
editor you create personalised reports and dashboards enabling you to zoom into the important details. It also
offers a unique commenting section in the sharing options which is great for collaboration purposes.