AI Methods in Data Warehousing
A System Architectural View
Walter Kriha
Business Driver: Customer Relationship
Management (CRM)
• learn more about your Customer
• Provide personalized offerings (cheaper, targeted)
• Make better use of in-house information (e.g. financial
research)
• Somehow use all the data collected
The web is accelerating the problems (terabytes of
clickstream data) and provides new solutions: Web-
mining, the Web-House)
CRM: Simulate Advisor Functions
Client oriented: Bank oriented:
• Know interests and • Know where to find
hobbies information and what
• Know personal situation applications to use
• Know situation in life
• Know how to
• Know plans and hopes
translate, summarize
and prepare for
customer
Plus: new ideas from automatic
• Know who to ask if in
knowledge discovery etc. that trouble
even a real advisor can’t do!
Overview
• Requirements coming from a dynamic,
personalized Portal Page
• Data Collection and DW Import
• AI Methods used to solve requirements
• How to flow the results back into the portal
A Portal: A self-adapting System
• Collect information for and about customers
• Learn from it
• Adapt to the individual customer by using the
“lessons learned”
The problem: a portal does not have the time to
learn. This needs to happen off-line in a warehouse!
DW Integration: Sources
Web Servers
Closed Application Servers
Loop
SAP
Content Ad Transaction Web Supplier IBM
Server Server Server Logs Extranet
Data Data Integration Platform Data
Marts Warehouse
RDBMS
PeopleSoft
e-Business Analytics Demographics/
External Sources
DW Integration: Structure
Mining Ware
Off-line tools house
Personalized
Rule Operational
information
and offerings Engine DB
Integ
Navigation,
Transactions, Log ration
Messages
Framewk Web stats
On-line External data
And Applications
What information do we have?
• The pages the customer selected (order, topics etc.)
• Customer interests from homepage self-configuration
• Customer transactions
• Customer messages (forum, advisor)
• Internal financial information
The data collection and import process needs to preserve the
links between different information channels (e.g. order of
customer activity)
Common: customize, filter, contact
Interest
Interest in ouretc.
inour
services
services
transactions
transactions (homepage
(homepage
Welcome [Link])
Rich,
config)
E-Banking: balance = We would like to point you to our
Interest
Interest
New in
inInstrument X that fits nicely
shares
shares
To etc.
etc. current investment strategy.
your
Portfolio: Siemens,
Swisskom, Esso, Message
Message
activity
activityCommon: Banner
Messages: 3 new
From foo: hi Mrs. Rich
Quotes: UBS 500, News: IBM invests in company Y
ARBA 200 Special
Special
interest forum
forum activity
activity
Links: [Link], interest
Research: asian equity update
UBS glossary etc. (filters
(filters
selected)
selected)
Forum: art banking, 12 new
Charts: Sony
What do we want to know?
• Does a customer know how to work the system (site
usability)?
• Does a customer voice dissatisfaction with company
(customer retention)
• If new financial information enters the system – which
customers might be interested in it (content extraction,
customer notification)?
Which AI techniques might answer those questions?
What do we want to provide?
• A personalized homepage that adapts itself to the customers
interests (from self-customization to automatic integration)
• An early warning system for disgruntled customers or
customers that have difficulties working the site
• An ontology for financial information
• An integrated view of the company and its services and
information (“electronic advisor”)
See: “Finance with a personal touch”, Communications of the
ACM Aug.2000/Vol.43 No.8
Common: customize, filter,Personal
contact etc.
Personal
Dynamic,
Dynamic, “touch”
“touch”
personalized
personalizedand
and Welcome Mrs. Rich,
INTEGRATED
INTEGRATED We would like to point you to our
homepage
Portfolio: Siemens,
homepage New Instrument X that fits nicely
add X? To your current investment strategy.
Messages: 3 new
From advisor: about X inv. Common: Banner about X
Quotes: UBS 500, News: IBM invests in company X,
Connect
Connect
X 100 X now listed on NASDAQ communities
communities
Research: X future prospects
and
Links: X homepage andsite
site
asian equity update content
[Link],. content
Forum: X is discussed here
Charts: X
Data Mining
• The automatic extraction of hidden predictive
information from large databases
• An AI-technique: automated knowledge
discovery, prediction and forensic analysis
through machine learning
Web Mining
• Adds text-mining, ontologies and things like xml
to the above
Data Mining Methods
Data mining
Data retained
Data Distilled
CBR. K-nearest n.
Equational Cross Tab Logical
Belief Nets Agents Decision Trees Rules
CART etc. GA Induct.
Neural Nets Statistics
Smooth surfaces Non-numeric data
Kohonen etc.
[Link] Non-symbolic results
Data Preparation
• Catch complete session data for a specific user
• Store meta-information from content with
behavioral data
• Create different data structures for different
analytics (e.g. Polygenesis)
Use a special log framework! Make sure there are meta-data for
the content available (e.g. dynamically generated page content)
Data Analysis
Content Mining (e.g Usage Mining (e.g.
Segmentation of Topics) Segmentation of Customers)
• Cluster Analysis • Pattern detection
• Classification • Association rules
Problem: How to express Problem: How to create a user
similarity and distance profile e.g from navigation data
•Linguistic analysis, statistics
(k-nearest-neighbours) collaborative filtering: derive
content similarities from
•Machine learning (Neuronal
behavioral similarities
nets, decision trees)
Example: Find Session Topics automatically
(Combined content and behavioral analysis)
• Use statistical cluster mining to extract page-views that co-
occur during sessions (visit coherence assumption)
• Use a concept learning algorithm that matches the clusters
(of page-views) with the meta-information of the pages to
extract common attributes
• Those common attributes form a “concept”
Learning Concepts
User A
Session flow
User B
Meta-Information
Conceptual
Learning Concept User Profile
Algorithm
The Text-Warehouse: Information Extraction
Financial Research
User profile Autom. IE Documents
With interests Database Tool (pdf, html, doc,xml)
Facts not Stories!
Serving personalized information requires fine-grained
extraction of interesting facts from text bodies in various
formats
Methods for Information Extraction
Natural Language Wrapper Induction
Processing
• Analyze Syntax to derive • Use contextual features to
Semantics infer semantics (e.g. html
• Context changes break tags)
algorithm • Very brittle in case of source
changes
Both methods use extraction patterns that were acquired
through machine learning based on training documents.
More textual methods
• Thematic Index: Generate the reference taxonomy
from training documents (linguistic and statistic
analysis)
• Clustering: group similar documents with respect
to a feature vector and similarity measure (SOM
and other clustering technologies)
Automatic Text Classification
Case: Building a directory for an enterprise portal
Rule based: Experts formulate rules and vertical
vocabularies (Verity, Intelligent Classifier)
Example-Based: A machine learning approach based
on training documents and iterative improvement (e.g
Autonomy, using Bayesian Networks)
Fully automated text classification is not feasible
today. Cyborg classification needed. More tagged data
needed.
The Meta-data/Ontology Problem
“The key limiting factor at present is the difficulty of
building and maintaining ontologies for web use”
[Link], Is there an Intelligent Agent in your
future?
This is also true for all kinds of information integration e.g.
financial research
The Solution: Semantic Web?
Agents and tools
use meta-data to
construct new
information
Logic, Rules etc.
Software
build, extracts Ontologies/Vocabularies
new Humans
Ontologies XML Schemas/RDF define
(e.g. meta-data
Ontobroker) and use
XML Syntax them
AI on Topic Maps?
Associations
Topics
Occurrences
See: James [Link], Ferrets and Topic Maps, Knowledge
Engineering for an Analytical Engine
Financial Research Integration
Dep.
[Link] Dep.
[Link]
Wrapper Induction
discovers facts
XML Editor Schema translation,
Warehouse
Meta- semantic
Data consistency checks
Topic e.g.
Maps Result DBs recommendations
Internal Information
Model Distribution users
Deployment
Mining Ware
Off-line tools house
Personalized Rule Operational
information Engine DB
and offerings
Rules (Profiles, Meta-
Data)
On-line
The Main Problems for the “Web-house”
Portal architecture must be designed to collect the
proper information and to use the results from the
web-house easily
Portal content is at the same time customer offer as
well as customer measuring tool
Few people understand both the portal system aspect
and the warehouse analytical aspect.
Resources
• Katherine [Link], Extracting • Information Discovery, A
Knowledge ( Characterization of Data
[Link]/feature Mining Technologies and
/010507/[Link] Process (
) [Link]/dm-tech.
• Dan Sullyvan, Beyond The htm
Numbers ( )
[Link]/ • Dan [Link], Data Mining
000410/[Link] on the Web
) ([Link]/archi
• Communications of the ACM, ves/2000/01/[Link])
August 2000/Vol.43 Nr. 8
Data Mining Tools (examples)
• IBM Intelligent Miner
• SPSS, Clementine
• SAS
• Netica (Belief Nets)