Embedding-Based Learning

From RNNs, LSTMs & Transformers to Human Understanding

Full conversation notes · June 2026

1. Research Patterns Found

Three core patterns across the research

The research builds a unified framework connecting AI architectures to human cognition. Every
section points to the same conclusion:

Knowledge = Concepts + Relationships. The relationships are what make knowledge

usable.

• Core thesis: Memory architecture → Learning strategy

• Key insight: Knowledge = Concepts + Relationships
• Practical goal: Build embeddings, not notes

The progression pattern (beginner → expert)

RNN thinking → Sequential processing (beginner default)
LSTM thinking → Selective memory with gates (intermediate)
Transformer thinking → Dense connection network (expert)

The deepest thread: embedding creation

The most original part of the research is the argument that human learning is fundamentally
about embedding creation, not fact storage. The calculus example illustrates this:

A novice has separate tokens: derivative, slope, tangent. An expert has one dense
embedding that activates all three simultaneously. That automatic co-activation IS the felt
experience of expertise.

2. What Embedding Creation Actually Means

When you learn a new concept, two completely different things can happen:
Fact storage (weak)
You add an entry: 'Backpropagation is the algorithm that computes gradients by traversing the
computational graph in reverse.' That sentence is stored. But it is a single isolated point. If
someone asks something slightly different, there is nothing to reach for.

Embedding creation (strong)

You reshape a landscape. The concept occupies a position defined entirely by its relationships
to everything else you know:
• Near chain rule (mathematical engine)
• Near gradient descent but distinct from it
• Connected to [Link]() in PyTorch
• Causal path to vanishing gradients → LSTM design
• Mirrors the reverse of forward pass
• Shares graph-traversal structure with Dijkstra and topological sort

That cluster of relationships IS your understanding of backprop. Not the definition.

The LLM analogy

LLMs learn word meanings by placing tokens in a high-dimensional vector space where position
is determined entirely by co-occurrence. 'King' is near 'Queen' not because someone labelled it
so, but because they appear in similar contexts. Your brain does something structurally
identical. The meaning emerges from the neighbourhood, not from the word itself.

3. The 6 Relationship Types

Every concept has 6 types of relationships that define its position in your mental map:

1. Similarity ↔
What is this close to? Cat ↔ Dog · Stack ↔ Queue · Array ↔ Linked List · Backprop ↔
Gradient descent

2. Contrast ≠
What is it frequently confused with, and where exactly do they diverge?
Backprop ≠ Gradient descent (backprop computes the gradient; GD uses it to update weights)
DFS ≠ BFS · Correlation ≠ Causation · Velocity ≠ Speed
 Contrast is not 'opposite.' It means: two things close enough to be confused, different in a
specific and important way. The gap must be small enough to matter.

3. Part–whole →
What is this a component of? Chain rule → Backprop · Node → Tree · Function → Program

4. Cause–effect ⇒
What does this produce? What produces it?
Loss function → Backprop → Weight updates → Parameter changes
Deep networks + sigmoid → Vanishing gradient → LSTM design

5. Generalization ⊃
Is this a special case of something? Does it generalize into variants?
Gradient descent ⊃ Adam, SGD, RMSProp · Autograd ⊃ Backprop · Graph ⊃ DAG ⊃
Computational graph

6. Transformation ⇄
What can this become? What can it transform into?
Backprop → Optimizer step (raw gradients become parameter changes)
Recursion ↔ DP · Brute force ↔ Optimized · Tree ↔ Graph

7 questions to ask for any new concept

1. What is it similar to?
2. What is it different from?
3. What is it made of?
4. What does it cause?
5. What causes it?
6. Where have I seen this pattern before?
7. What can it transform into?

4. The 4 Cases When a Concept Feels Isolated

Case 1 — You don't know the neighbours yet

The concept is not isolated. Your map is incomplete. Relationships require two endpoints. If you
haven't learned adjacent concepts yet, edges are invisible even though they exist.
 Fix: Keep learning forward. Return to this concept after building more surrounding territory.
The edges will appear on their own.

Case 2 — You're only searching for one relationship type

Most people default to similarity. But a concept might have no strong siblings and still have rich
cause-effect, part-whole, or transformation edges. Run through all 6 types deliberately.
Example: The computational graph has no obvious similarity. But it has a powerful part-
whole relationship (backprop lives inside it), a transformation relationship (chain rule applied
at every node), and a generalization relationship (it's a specialised DAG).

Case 3 — It's a primitive concept

Some concepts are bedrock. They don't have rich lateral relationships because they're not
derived from anything simpler. Their connections all point outward into things they generate.
Example: 'A node in a graph' has almost no similarity relationships. But everything in graph
theory is built from it. The embedding is a root, not a hub.
Fix: Map outward instead of sideways. Ask: what exists only because this concept exists?

Case 4 — Genuinely new territory, no landing pad

This is the RNN problem stated exactly. The concept arrives with no surrounding context.
There's nothing nearby to attach it to, so no embedding can form no matter how many times you
read the definition.
Example: Recursion feels like noise to someone with no CS background — until you show them
Russian nesting dolls. One concrete analogy creates one edge, and the whole thing becomes
learnable.
Fix: Before studying the new concept, first study something adjacent you already
understand. Build a landing pad. Then introduce the concept and immediately force one
connection. A single edge to familiar territory is enough.

5. What Contrast Really Means

Contrast is the most underrated and most misunderstood of the 6 relationship types.

It is NOT about opposites. It is about:

Two things that are close enough to be confused, but different in a specific and important
way. The gap has to be small enough to matter.
A contrast edge tells you three things:

1. What they share — why the confusion is understandable in the first place.
Backprop and gradient descent are both part of the training loop, both involve gradients, both
point toward minimizing loss. Of course people blur them.

2. Where exactly they diverge — the single specific point of difference.

Backprop computes the gradient. Gradient descent uses it to move the parameters. One is
differentiation, the other is optimization. Different operations, different concerns, happen to be
used together.

3. When the distinction matters — where conflating them breaks something.

If you think backprop and gradient descent are the same, you can't reason about why Adam
outperforms SGD (that's a GD question, unrelated to backprop), or why backprop fails in deep
nets even with a good optimizer (that's a vanishing gradient question, unrelated to the
optimizer).

A contrast edge is not just 'these two are different.' It's a precisely located boundary with a
specific reason — and that boundary is often where the deepest understanding lives.

6. The 5-Step Learning Process

Step 1 — Define (5 min) · RNN phase

Write the definition in your own words. If you need jargon, you're not ready to move on.
Ask: What is it? What category does it belong to?
Output: one clear sentence, no bullet points.

Step 2 — Connect (15 min) · LSTM input gate

Map at least 3 of the 6 relationship types. The minimum viable embedding needs similarity,
contrast, and one cause-effect edge.
Ask: What is it like? What is it unlike? What does it cause? What is it part of?
Output: a small hand-drawn node map with 5 or more edges.

Step 3 — Contrast (10 min) · LSTM forget gate

Name what this concept is most commonly confused with. Write 2–3 explicit 'X ≠ Y because…'
statements.
Ask: When would I wrongly use this? What breaks it? What does it look like but isn't?
 Output: a 2–3 item not-list — the forget gate of your embedding.

Step 4 — Apply (20–30 min) · Transformer attention

Solve one real problem that forces the concept to activate. Then explain it out loud as if to a
junior.
Ask: Can I derive it from scratch? Can I use it without looking anything up?
Output: a solved problem or a 5-line explanation you'd stand behind.

Step 5 — Retrieve (10 min · repeat on days 1, 3, 7, 14) · Spaced retrieval

Close everything. Redraw the node map from memory. Missing edges are your next revision
targets, not signs of failure.
Ask: Which connections feel weak? What new edges have appeared since last time?
Output: a reconstructed map. Gaps = the actual work remaining.

7. Daily 60-Minute Session Template

0–10 min · Retrieve

Reconstruct yesterday's concept map cold, before opening any notes.

10–25 min · Define + Connect

Steps 1–2 on today's new concept.

25–35 min · Contrast

Write the not-list. Find the failure mode and the common confusion.

35–55 min · Apply

One problem or teach-back exercise. No notes.

55–60 min · Close the loop

Add new edges to today's map. Note gaps for tomorrow's retrieval session.

8. Golden Rules
8. Never move to step 2 until you can say step 1 without notes. The definition is the anchor
— a weak anchor means every edge you add will drift.
9. Rereading feels like learning but isn't. Recognition is not retrieval. If you can follow along
but can't reconstruct, you have exposure, not an embedding.
10. One strong cross-domain connection beats ten same-domain connections. Backprop ↔
topological sort (DAG traversal) is more valuable than backprop ↔ gradient descent,
because it fires from an unexpected direction in an interview or real problem.
11. The question to end every session: 'If I saw this concept in disguise — different notation,
different domain — would I recognise it?' If not, the embedding isn't dense enough yet.
12. The session should always start with retrieval, not input. Most people open their notes
first. Open a blank page and reconstruct — only then check what's missing.

Embedding-Based Learning · Conversation notes · June 2026