Dbms - Module 5
Dbms - Module 5
Data Warehouse:
A data warehouse is a subject-oriented, integrated, time-variant and non-volatile collection
of data in support of management's decision making process.
Subject-Oriented: A data warehouse can be used to analyze a particular subject area.
For example, "sales" can be a particular subject.
Integrated: A data warehouse integrates data from multiple data sources. For example,
source A and source B may have different ways of identifying a product, but in a data
warehouse, there will be only a single way of identifying a product.
Time-Variant: Historical data is kept in a data warehouse. For example, one can
retrieve data from 3 months, 6 months, 12 months, or even older data from a data
warehouse. This contrasts with a transactions system, where often only the most
recent data is kept. For example, a transaction system may hold the most recent
address of a customer, where a data warehouse can hold all addresses associated with
a customer.
Non-volatile: Once data is in the data warehouse, it will not change. So, historical data
in a data warehouse should never be altered.
Tier-1: The bottom tier is a warehouse database server that is almost always a relational
database system. Back-end tools and utilities are used to feed data into the bottom tier
from operational databases or other external sources (such as customer profile information
provided by external consultants). These tools and utilities perform data extraction,
cleaning, and transformation (e.g., to merge similar data from different sources into a
unified format), as well as load and refresh functions to update the data warehouse . The
data are extracted using application program interfaces known as gateways. A gateway is
supported by the underlying DBMS and allows client programs to generate SQL code to be
executed at a server. Examples of gateways include ODBC (Open Database Connection) and
OLEDB (Open Linking and Embedding for Databases) by Microsoft and JDBC (Java Database
Connection). This tier also contains a metadata repository, which stores information about
the data warehouse and its contents.
2
DBMS - MODULE - V - ADVANCED TOPICS
Tier-2: The middle tier is an OLAP server that is typically implemented using either a
relational OLAP (ROLAP) model or a multidimensional OLAP. OLAP model is an extended
relational DBMS that maps operations on multidimensional data to standard relational
operations.
A multidimensional OLAP (MOLAP) model, that is, a special-purpose server that directly
implements multidimensional data and operations.
Tier-3: The top tier is a front-end client layer, which contains query and reporting tools,
analysis tools, and/or data mining tools (e.g., trend analysis, prediction, and so on).
2. Data mart: A data mart contains a subset of corporate-wide data that is of value to a
specific group of users. The scope is confined to specific selected subjects. For example, a
marketing data mart may confine its subjects to customer, item, and sales. The data
contained in data marts tend to be summarized. Data marts are usually implemented on
low-cost departmental servers that are UNIX/LINUX- or Windows-based. The
implementation cycle of a data mart is more likely to be measured in weeks rather than
months or years. However, it may involve complex integration in the long run if its design
and planning were not enterprise-wide.
Depending on the source of data, data marts can be categorized as independent or
dependent. Independent data marts are sourced from data captured from one or more
operational systems or external information providers, or from data generated locally
within a particular department or geographic area. Dependent data marts are sourced
directly from enterprise data warehouses.
3
DBMS - MODULE - V - ADVANCED TOPICS
Data Mining :
Given databases of sufficient size and quality, data mining technology can generate new
business opportunities by providing these capabilities:
5
DBMS - MODULE - V - ADVANCED TOPICS
1. Knowledge Base: This is the domain knowledge that is used to guide the search or
evaluate the interestingness of resulting patterns. Such knowledge can include
concept hierarchies, used to organize attributes or attribute values into different
levels of abstraction. Knowledge such as user beliefs, which can be used to assess a
pattern’s interestingness based on its unexpectedness, may also be included. Other
examples of domain knowledge are additional interestingness constraints or
thresholds, and metadata (e.g., describing data from multiple heterogeneous
sources).
2. Data Mining Engine: This is essential to the data mining system and ideally consists
of a set of functional modules for tasks such as characterization, association and
correlation analysis, classification, prediction, cluster analysis, outlier analysis, and
evolution analysis.
3. Pattern Evaluation Module: This component typically employs interestingness
measures interacts with the data mining modules so as to focus the search toward
interesting patterns. It may use interestingness thresholds to filter out discovered
patterns. Alternatively, the pattern evaluation module may be integrated with the
mining module, depending on the implementation of the data mining method used.
For efficient data mining, it is highly recommended to push the evaluation of pattern
interestingness as deep as possible into the mining processor as to confine the
search to only the interesting patterns.
4. User interface: This module communicates between users and the data mining
system, allowing the user to interact with the system by specifying a data mining
query or task, providing information to help focus the search, and performing
exploratory data mining based on the intermediate data mining results. In addition,
this component allows the user to browse database and data warehouse schemas or
data structures, evaluate mined patterns, and visualize the patterns in different
forms.
6
DBMS - MODULE - V - ADVANCED TOPICS
Knowledge Discovery in Databases (KDD)
Some people treat data mining same as Knowledge discovery while some people view data mining
essential step in process of knowledge discovery.
Here is the list of steps involved in knowledge discovery process:
Data Cleaning - In this step the noise and inconsistent data is removed.
Data Integration - In this step multiple data sources are combined.
Data Selection - In this step relevant to the analysis task are retrieved from the database.
Data Transformation - In this step data are transformed or consolidated into forms
appropriate for mining by performing summary or aggregation operations.
Data Mining - In this step intelligent methods are applied in order to extract data patterns.
Pattern Evaluation - In this step, data patterns are evaluated.
Knowledge Presentation - In this step, knowledge is represented.
7
DBMS - MODULE - V - ADVANCED TOPICS
class Dbms()
Creates a Dbms object. Dbms objects support basic operations on a database.
begin()
Closes the database object. It is safe to try to close a database object even if it is not open.
commit()
Commits the current transaction.
compact()
Compacts the database, reclaiming unused space in the database file. create(dbname)
Executes an SQL query. On success, returns 0 if a DDL (SQL schema update) statement was
executed. Returns the number of rows inserted, updated, or deleted, if a DML (SQL data
update) statement was executed.
8
DBMS - MODULE - V - ADVANCED TOPICS
open(dbname)
Opens the database in file dbname. This should be a full Unicode path name, for example,
u'c:\\[Link]'.
rollback()
xml
Copy code
<bookstore>
<book>
<title>Harry Potter</title>
<author>J.K. Rowling</author>
</book>
<book>
<title>The Lord of the Rings</title>
<author>J.R.R. Tolkien</author>
</book>
</bookstore>
XML databases provide a specialized solution for storing and querying XML data
efficiently. Understanding the hierarchical nature of XML data and leveraging
appropriate query languages are essential for effective use of XML databases in various
applications.
11
DBMS - MODULE - V - ADVANCED TOPICS
Cloud based systems
Cloud-based systems refer to computing systems and services that are delivered over the
internet. Instead of storing data or running applications on a local physical server or
computer, cloud computing allows users to access resources such as storage, processing
power, and software applications remotely, typically via a web browser or specialized
client application.
Cloud-based systems offer numerous benefits, including cost savings, scalability, flexibility,
and improved collaboration and accessibility. However, they also raise concerns around
data security, privacy, and vendor lock-in, which organizations must address when
adopting cloud technologies.
12
DBMS - MODULE - V - ADVANCED TOPICS
NoSQL, which stands for "Not Only SQL," is a term used to describe a variety of database
technologies that are designed to handle large volumes of unstructured or semi-structured
data. Unlike traditional relational databases (SQL databases), NoSQL databases are typically
schema-less, meaning they don't require a fixed schema and can accommodate flexible data
models.
1. Flexible Data Models : NoSQL databases can handle various types of data, including
structured, semi-structured, and unstructured data. They can store documents, key-value
pairs, column-family data, and graphs, among other formats.
2. Scalability : NoSQL databases are designed to scale horizontally, meaning they can
easily distribute data across multiple servers or nodes in a cluster. This allows them to
handle large amounts of data and high read/write loads.
3. High Performance : NoSQL databases are optimized for specific use cases, such as high-
speed reads or writes, and can often outperform traditional SQL databases in these
scenarios.
4. No Schema Enforcement : Unlike SQL databases, which enforce a rigid schema, NoSQL
databases typically do not enforce a fixed schema. This allows for greater flexibility in data
modeling and schema evolution.
5. CAP Theorem : NoSQL databases are often designed with the CAP theorem in mind,
which states that it's impossible for a distributed system to simultaneously guarantee
Consistency, Availability, and Partition tolerance. NoSQL databases typically prioritize
either consistency and availability (CA systems) or consistency and partition tolerance (CP
systems), depending on the specific requirements of the application.
6. Types of NoSQL Databases :
- Document Stores : Examples include MongoDB, Couchbase, and CouchDB. They store
data as JSON-like documents.
- Key-Value Stores : Examples include Redis, Amazon DynamoDB, and Riak. They store
data as key-value pairs.
- Column-Family Stores : Examples include Apache Cassandra and HBase. They store
data in columns rather than rows.
- Graph Databases : Examples include Neo4j, Amazon Neptune, and JanusGraph. They
are optimized for storing and querying graph data.
7. Use Cases : NoSQL databases are commonly used in scenarios where traditional SQL
databases may struggle, such as big data analytics, real-time web applications, content
management systems, IoT data management, and social networks.
Overall, NoSQL databases offer flexibility, scalability, and performance advantages for
handling large volumes of unstructured or semi-structured data in modern applications.
However, it's essential to choose the right NoSQL database based on the specific
requirements and characteristics of your application.
13
DBMS - MODULE - V - ADVANCED TOPICS
Hbase data model
1. Tables: In HBase, data is organized into tables. Each table consists of rows and
columns. Tables in HBase are sparse, meaning they can have a large number of rows,
but not all rows need to contain data for every column.
2. Rows: Rows in HBase are uniquely identified by a row key, which is a byte array. The
row key is used to retrieve and store data efficiently. Rows are sorted lexicographically
based on their row keys, which allows for efficient range scans.
3. Columns and Column Families: Columns in HBase are grouped into column families.
Each column family is a logical grouping of columns and is stored together on disk.
When you create a table in HBase, you must specify one or more column families.
Unlike traditional relational databases, HBase does not require you to define column
names upfront; instead, columns are defined dynamically as data is inserted into the
table.
4. Column Qualifiers: Within each column family, columns are identified by a column
qualifier, which is a byte array. Column qualifiers are used to distinguish different
columns within a column family.
5. Cells: The combination of a row key, column family, column qualifier, and timestamp
uniquely identifies a cell in HBase. Cells are the basic unit of data storage in HBase and
can contain a value along with a timestamp indicating when the value was last
modified.
6. Timestamps: Each cell in HBase can have multiple versions, each with its own
timestamp. This allows you to store and retrieve historical versions of data. By default,
HBase keeps all versions of a cell, but you can configure it to retain only a certain
number of versions or versions within a specific time range.
7. Table Schema Evolution: HBase allows for schema evolution, meaning you can add or
remove column families and columns from a table without needing to modify existing
data. This flexibility makes it easy to adapt the data model to changing requirements
over time.
8. Secondary Indexes: HBase does not support secondary indexes out of the box.
However, you can implement secondary indexes using techniques such as maintaining
separate tables or using Apache Phoenix, which provides SQL-like querying capabilities
on top of HBase and supports secondary indexes.
Overall, the HBase data model is optimized for storing and retrieving large volumes of
structured data efficiently in a distributed environment. It provides flexibility, scalability,
and high availability, making it suitable for a wide range of use cases, including real-time
analytics, time-series data storage, and serving as a backend for web applications.
14
DBMS - MODULE - V - ADVANCED TOPICS
Database Tuning
1. Introduction to Database Tuning:
- Definition: Database tuning refers to the process of optimizing the performance of a
database.
- Importance: Efficient database performance is crucial for ensuring that applications run
smoothly and respond quickly to user queries.
4. Query Optimization:
- Query structure: Optimizing query structure to minimize resource usage.
- Indexing: Properly indexing tables to speed up data retrieval.
- Joins: Optimizing join queries to reduce execution time.
- Use of hints: Providing hints to the query optimizer to improve execution plans.
5. Indexing Strategies:
- Types of indexes: B-tree, hash, bitmap, etc.
- Index selection: Choosing appropriate columns for indexing based on query patterns.
- Index maintenance: Regularly updating and maintaining indexes to ensure optimal
performance.
These are some of the key topics that might be covered in a lecture on database tuning.
Each topic can be explored in further detail based on the specific needs and level of the
audience.
16
DBMS - MODULE - V - ADVANCED TOPICS
Case Study for Design and Manage the Database for any Project.
AIM: A publishing company produces scientific books on various subjects. The books
are written by authors who specialize in one particular subject. The company
employs editors who, not necessarily being specialists in a particular area, each take
sole responsibility for editing one or more publications.
LAB ASSIGNMENT:
1. Analyze the data required.
2. Normalize the attributes.
3. Create the logical data model using E-R diagrams
17
DBMS - MODULE - V - ADVANCED TOPICS
LAB ASSIGNMENT:
1. Analyze the data required.
2. Normalize the attributes.
3. Create the logical data model using E-R diagrams
18
DBMS - MODULE - V - ADVANCED TOPICS
AIM: A database is to be designed for a Car Rental Co. (CRC). The information required
includes a description of cars, subcontractors (i.e. garages), company expenditures,
company revenues and customers. Cars are to be described by such data as: make, model,
year of production, engine size, and fuel type, number of passengers, registration number,
purchase price, purchase date, rent price and insurance details. It is the company policy
not to keep any car for a period exceeding one year.
All major repairs and maintenance are done by subcontractors (i.e. franchised garages),
with whom CRC has long-term agreements. Therefore the data about garages to be kept in
the database includes garage names, addresses, range of services and the like. Some
garages require payments immediately after a repair has been made; with others CRC has
made arrangements for credit facilities. Company expenditures are to be registered for all
outgoings connected with purchases, repairs, maintenance, insurance etc.
Similarly the cash inflow coming from all sources - car hire, car sales, insurance claims -
must be kept of [Link] maintains a reasonably stable client base. For this privileged
category of customers special credit card facilities are provided. These customers may
also book in advance a particular car. These reservations can be made for any period of
time up to one month. Casual customers must pay a deposit for an estimated time of
rental, unless they wish to pay by credit card. All major credit cards are accepted. Personal
details (such as name, address, telephone number, driving license, number) about each
customer are kept in the database.
LAB ASSIGNMENT:
1. Analyze the data required.
2. Normalize the attributes.
3. Create the logical data model using E-R diagrams
19
DBMS - MODULE - V - ADVANCED TOPICS
LAB ASSIGNMENT:
1. Analyze the data required.
2. Normalize the attributes.
3. Create the logical data model using E-R diagrams
---------------------------------------------------------------------------------------------------
20