The GenAI Stack
Andreas Kollegger of Neo4j, a database company
GenerativeAI
2 Neo4j Inc. All rights reserved 2023
GenerativeAI
is a Parrot
3 Neo4j Inc. All rights reserved 2023
GenerativeAI
is a Sock Puppet
4 Neo4j Inc. All rights reserved 2023
GenerativeAI
is Alien Technology
5 Neo4j Inc. All rights reserved 2023
GenerativeAI
● Learns random sentences
from random people
● Talks like a person but doesn't really
understand what it's saying
● Occasionally speaks absolute non sense
● Sensitive to question phrasing
● Answers reflect the person asking
● Can't explain or verify answers
● Limited to public "knowledge"
6 Neo4j Inc. All rights reserved 2023
GenerativeAI
● Learns random sentences
from random people
● Talks like a person but doesn't really
understand what it's saying
● Occasionally speaks absolute non sense
● Sensitive to question phrasing
● Answers reflect the person asking
● Can't explain or verify answers
● Limited to public "knowledge"
7 Neo4j Inc. All rights reserved 2023
How do we
integrate
with the alien
technology?
8 Neo4j Inc. All rights reserved 2023
Generative AI is a new layer in the Stack
Application
GenAI
Database
Files
9 Neo4j Inc. All rights reserved 2023
GenAI Stack with Neo4j
Application LangChain
GenAI an "orchestration
framework" for
integrating with LLMs
Database
Files
10 Neo4j Inc. All rights reserved 2023
GenAI Stack with Neo4j
ollama
GenAI locally managed
language models
Database
Files
11 Neo4j Inc. All rights reserved 2023
GenAI Stack with Neo4j
Neo4j
knowledge graph to
augment the LLM
Database
Files
12 Neo4j Inc. All rights reserved 2023
GenAI Stack with Neo4j
import
sample data sources for
constructing a
knowledge graph
Files
13 Neo4j Inc. All rights reserved 2023
GenAI Stack with Neo4j
LangChain GenAI Stack
A reference
implementation of
Retrieval Augmented
Ollama Generation
Neo4j
Files
14 Neo4j Inc. All rights reserved 2023
Retrieval Augmented
Generation (RAG)
15 Neo4j Inc. All rights reserved 2023
Retrieval Augmented Generation
A Generative AI
app uses an
LLM to provide
answers
(aka ChatGPT)
16 Neo4j Inc. All rights reserved 2023
Retrieval Augmented Generation
RAG augments
the LLM
by making
requests to a
database
17 Neo4j Inc. All rights reserved 2023
Three Data Access Scenarios for RAG
1) Pure Text 2) Mixed Text + Data 3) Pure Data
unstructured data structured data with structured data with
long-form text short text values
typically PDFs or other database used for CMS general database
text documents
18 Neo4j Inc. All rights reserved 2023
Three Data Access Scenarios for RAG
1) Pure Text 2) Mixed Text + Data 3) Pure Data
unstructured data structured data with structured data with
long-form text short text values
typically PDFs or other database used for CMS general database
text documents
19 Neo4j Inc. All rights reserved 2023
Three Data Access Scenarios for RAG
1) Pure Text 2) Mixed Text + Data 3) Pure Data
unstructured data structured data with structured data with
long-form text short text values
typically PDFs or other database used for CMS general database
text documents
20 Neo4j Inc. All rights reserved 2023
Three Data Access Scenarios for RAG
1) Pure Text 2) Mixed Text + Data 3) Pure Data
unstructured data structured data with structured data with
long-form text short text values
typically PDFs or other database used for CMS general database
text documents
21 Neo4j Inc. All rights reserved 2023
Three Data Access Scenarios for RAG
1) Pure Text 2) Mixed Text + Data 3) Pure Data ?
unstructured data structured data with structured data with
long-form text short text values
typically PDFs or other database used for CMS general database
text documents
22 Neo4j Inc. All rights reserved 2023
Three Data Access Scenarios for RAG
1) Pure Text 2) Mixed Text + Data 3) Pure Data
unstructured data structured data with structured data with
long-form text short text values
typically PDFs or other database used for CMS general database
text documents
23 Neo4j Inc. All rights reserved 2023
Text is the coupling to the
Alien Technology
24 Neo4j Inc. All rights reserved 2023
3) Pure Data Retrieval
25 Neo4j Inc. All rights reserved 2023
Retrieval Augmented Generation
Pure Data Retrieval
User
1. accept prompt from user 1
Prompt Response
2. pass prompt to a fine-tuned code
generation model 2
Text-to-Query
5
LLM
3. run query against database LLM Chat API
4. combine user prompt with query
3 4 Prompt +
Database Relevant
results Search Information
5. generate natural language results
with LLM Database Relevant Results
26 Neo4j Inc. All rights reserved 2023
3) Pure Data Retrieval Challenges
● Getting it to work at all – generating syntactically correct queries
● Getting it to do the right thing – producing meaningful results
● Avoiding accidents – mistaken deletion
● Preventing malicious intent – SQL injection gone wild
An area of active research and development.
27 Neo4j Inc. All rights reserved 2023
1) Pure Text Retrieval
28 Neo4j Inc. All rights reserved 2023
Retrieval Augmented Generation
Pure Text Retrieval
User
1. accept prompt from user 1
Prompt Response
2. generate an embedding for the
prompt 2 5
Embedding API LLM
3. perform a vector/similarity Chat API
search on the embedding
3 4 Prompt +
4. combine user prompt with search Database Relevant
Search Information
results
5. generate natural language results Relevant Results
Database
with LLM
29 Neo4j Inc. All rights reserved 2023
Pure Text Preparation
1. Chunking 2. Embedding 3. Persistence
Lorem ipsum dolor sit
[0.2,0.2,0.1,0.7]
amet, consectetur
adipiscing elit, sed do
[0.3,0.2,0.1,0.5]
eiusmod tempor
incididunt ut labore et
[0.4,0.2,0.1,0.7]
dolore magna aliqua. Ut
enim ad minim veniam,
[0.5,0.2,0.1,0.7]
quis nostrud
exercitation ullamco
[0.6,0.2,0.1,0.7]
laboris nisi ut aliquip
30 Neo4j Inc. All rights reserved 2023
Pure Text, Just Chunks Just Chunks
How?
● pick a chunk method & size
● each chunk is a record Lorem ipsum dolor sit
amet, consectetur
● store chunk with metadata adipiscing elit, sed do
eiusmod tempor
incididunt ut labore et
Challenges: dolore magna aliqua. Ut
● what makes a good chunk?
however, never do this…
● potential chunk duplication
exercitation ullamco
● how to re-assemble chunk laboris nisi ut aliquip
context? adipiscing elit, sed do
eiusmod tempor
31 Neo4j Inc. All rights reserved 2023
Pure Text, Just Chunks Just Chunks
How?
● pick a chunk method & size
● each chunk is a record Lorem ipsum dolor sit
amet, consectetur
● store chunk with metadata adipiscing elit, sed do
eiusmod tempor
incididunt ut labore et
Challenges: dolore magna aliqua. Ut
● what makes a good chunk?
however, never do this…
● potential chunk duplication
exercitation ullamco
● how to re-assemble chunk laboris nisi ut aliquip
context? adipiscing elit, sed do
eiusmod tempor
32 Neo4j Inc. All rights reserved 2023
Pure Text, Parent-Child Parent-Child Chunks
How?
Lorem ipsum dolor sit
● connect each chunk to original amet, consectetur
document
● connect previous/next chunk adipiscing elit, sed do
eiusmod tempor
Challenges:
incididunt ut labore et
● what about cross-document dolore magna aliqua. Ut
chunks?
● explaining the relevance
however, never do this…
exercitation ullamco
laboris nisi ut aliquip
33 Neo4j Inc. All rights reserved 2023
Explicit Similarity Ontology
How?
● Thing
● named entity recognition
● metadata extraction Lorem tempor nostrud
● document cross-linking,
page ranking
Lorem ipsum dolor Lorem ipsum dolor
adipiscing elit, sed enim ad minim
sit amet, sit amet,
do eiusmod tempor veniam, quis nostrud
Challenges: consectetur consectetur
● does chunk similarity provide the
most relevant answer?
● what about the person asking the
question?
34 Neo4j Inc. All rights reserved 2023
Pure Text, Context
Context
How?
● named entity recognition
● metadata extraction Lorem tempor nostrud
● document cross-linking,
page ranking
Lorem ipsum dolor Lorem ipsum dolor
adipiscing elit, sed enim ad minim
sit amet, sit amet,
do eiusmod tempor veniam, quis nostrud
consectetur consectetur
Challenges:
● does chunk similarity provide the
most relevant answer?
● what about the person asking the
question?
35 Neo4j Inc. All rights reserved 2023
Text in, find relevant text,
return text
Lorem ipsum dolor sit amet, consectetur
Lorem ipsum dolor sit amet, consectetur
Lorem ipsum Lorem ipsum dolor sit amet, consectetur
Lorem ipsum dolor sit amet, consectetur
36 Neo4j Inc. All rights reserved 2023
2) Mixed Text & Data
Retrieval
37 Neo4j Inc. All rights reserved 2023
Retrieval Augmented Generation
Mixed Text & Data
Retrieval User
1
Prompt Response
1. accept prompt from user
2 5
2. generate an embedding for the LLM
Embedding API
prompt Chat API
3. perform a data query,
anchored on the embedding 3
Database
4 Prompt +
Relevant
4. combine user prompt with search Search Information
results
5. generate natural language results
Database Relevant Results
with LLM
38 Neo4j Inc. All rights reserved 2023
Mixed Text & Data
Mixed Text & Data
How?
● extracted information Lorem tempor nostrud
● application data
● user data
Lorem ipsum dolor Lorem ipsum dolor
● explicit semantic connections
adipiscing elit, sed enim ad minim
sit amet, sit amet,
do eiusmod tempor veniam, quis nostrud
consectetur consectetur
Challenges:
● what is most relevant,
to the user?
39 Neo4j Inc. All rights reserved 2023
Text plus data for complete,
relevant answers
Lorem ipsum dolor sit amet, consectetur
Andreas Kollegger
Lorem ipsum Cambridge, UK
Gardening
40 Neo4j Inc. All rights reserved 2023
Knowledge Graphs
41 Neo4j Inc. All rights reserved 2023
A Knowledge Graph is a structured way of representing
information, typically using nodes and edges to depict
relationships between entities (e.g., people, places,
things, concepts).
These entities and their interconnections form a
graph-like structure, which can be used to model
complex sets of data and the relationships within that
data.
– ChatGPT
42 Neo4j Inc. All rights reserved 2023
In knowledge representation and reasoning, a Knowledge
Graph is a knowledge base that uses a graph-structured
data model or topology to represent and operate on data.
Knowledge graphs are often used to store interlinked
descriptions of entities – objects, events, situations or
abstract concepts – while also encoding the semantics or
relationships underlying these entities.[1]
– Wikipedia
43 Neo4j Inc. All rights reserved 2023
A Knowledge Graph is a data structure
where information is stored
in both objects and
the relationships between objects.
– Andreas Kollegger
44 Neo4j Inc. All rights reserved 2023
For example,
StackOverflow as a Knowledge Graph
45 Neo4j Inc. All rights reserved 2023
Data starts with things.
In StackOverflow
those are… Question
Tag User
Answer
46 Neo4j Inc. All rights reserved 2023
Data records information
about things
title: string
Question body: string
link: string
Tag name: string User name: string
reputation: integer
Answer body: string
accepted: boolean
47 Neo4j Inc. All rights reserved 2023
Data records can be related
Question
TAGGED@ ASKED@
Tag ANSWERS@ User
PROVIDED@
Answer
48 Neo4j Inc. All rights reserved 2023
StackOverflow
Knowledge Graph
49 Neo4j Inc. All rights reserved 2023
Data relationships create patterns
Question
TAGGED@ ASKED@
Tag ANSWERS@ User
PROVIDED@
Answer
50 Neo4j Inc. All rights reserved 2023
Data pattern: from users to questions
Question
TAGGED@ ASKED@
Tag ANSWERS@ User
PROVIDED@
Answer
(User)-[ASKED]->(Question)
51 Neo4j Inc. All rights reserved 2023
Data pattern: from users to answers
Question
TAGGED@ ASKED@
Tag ANSWERS@ User
PROVIDED@
Answer
(User)-[PROVIDED]->(Answer)
52 Neo4j Inc. All rights reserved 2023
Data pattern: from answers to tagged questions
Question
TAGGED@ ASKED@
Tag ANSWERS@ User
PROVIDED@
Answer
(Answer)-[ANSWERS]->(Question)-[TAGGED]->(Tag)
53 Neo4j Inc. All rights reserved 2023
Knowledge Graph
Facts about people, places, or things interlinked by their
relationships
Human and LLM-friendly readable format
Organizing principle provides context for reasoning
about the data
Making it Real
understand customer behavior and preferences
better, to provide personalized services.
GenAI Recommendations
Apps in the Customer Service
Enterprise
process documents to accelerate and
reduce the effort to codify rules
quickly answer customer
questions from thousands of
pages of policy documentation identify bottlenecks in bill of materials
Supply Chain to support demand for customers
product recall and associated quality control checking
Knowledge Base patient portal transformation initiative
internal documentation search
answer prospect questions on the fly
chatbot which queries publications,
Policy & Pricing news, etc. by predefined prompts.
simplify and summarize resources to help
technicians resolve errors on the factory floor.
identify the right coverage, with the least effort, at
the best possible price.
Fraud Detection
Improve Answers: Pure text w/ Knowledge Graph
Global Equipment Manufacturer:
Challenge: Support agents want to lower their Response
Support Agent
mean time to repair (MTTR) by quickly
providing a resolution, but the best answer is
hidden among thousands of field docs,
engineering summaries, documentation, and Prompt LLM
Chat API
case histories.
LLM
Embeddings API
Solution: Use vector embeddings to look for Knowledge
the most relevant information, while the graph of
Neo4j knowledge graph adds the exact match Embedding
equipment
for critical components.
Vector
Search in
Desired Outcome: Reduce MTTR, and provide Relevant Results
Vector
the best option for the customer quickly and / Documents
Index
with confidence.
U Knowledge Access: Pure data w/ Text to Query
Prompt (Description
Global Electronics Manufacturer: of Desired Products)
Seller
Challenge: Sellers want to quote the right
product combination and price for every
opportunity across five unique types of Response LLM API
external stakeholders. Information is stored
across millions of bills of materials and
product combinations.
LLM API Cypher Query
Solution: Bring bills of materials into a
knowledge graph that can be queried by an
LLM interface by sellers. Relevant Results
Bill of Materials
Desired Outcome: Quote the best option for Knowledge
each customer quickly and with confidence. Graph
Complex Processes: Mixed Text + Data
Government Procurement Entity:
Supplier
Challenge: Government entities can overspend or
acquire goods and services redundantly because Procurement Officer
Enterprise App
the volume of RFPs makes reviewing each of them
Ingests New RFP
resource prohibitive.
User is Emailed with
Solution: Use LLM to read the nature of RFP and Recommendation
classify it accordingly. Compare the new RFP LLM API
against the knowledge graph of active and historic Categorizes RFP
RFPs and associated spend. Recommend LLM API
opportunities for consolidation or terms negotiation Generates
with suppliers. personalized text
Knowledge
Desired Outcome: Save tax dollars for other graph of RFPs
important projects and spending Predefined Prompt
patterns with Results
RAG is the way to bridge
GenAI with Business Data
Lorem ipsum dolor sit amet, consectetur
Andreas Kollegger
Lorem ipsum Cambridge, UK
Gardening
60 Neo4j Inc. All rights reserved 2023
Thanks!
@akollegger most places
andreas@[Link]