A standalone, framework-agnostic semantic cache for LLM applications backed by Valkey. Uses Valkey's vector search (valkey-search module) for similarity matching with built-in OpenTelemetry tracing and Prometheus metrics. Full adapter parity with @betterdb/agent-cache.

• Valkey 8.0+ with the valkey-search module loaded
• Or Amazon ElastiCache for Valkey (8.0+)
• Or Google Cloud Memorystore for Valkey
• Node.js >= 20.0.0

npm install @betterdb/semantic-cache iovalkey

iovalkey is a required peer dependency.

The only semantic cache library that is simultaneously Valkey-native (explicit handling of valkey-search API differences), standalone (no coupling to any AI framework), and has built-in OpenTelemetry + Prometheus instrumentation at the cache operation level.

import Valkey from 'iovalkey';
import { SemanticCache } from '@betterdb/semantic-cache';
import { createOpenAIEmbed } from '@betterdb/semantic-cache/embed/openai';

const client = new Valkey({ host: 'localhost', port: 6399 });

const cache = new SemanticCache({
  client,
  embedFn: createOpenAIEmbed(), // or createVoyageEmbed(), createOllamaEmbed(), etc.
  defaultThreshold: 0.15,       // loosen slightly to catch paraphrases with high confidence
  defaultTtl: 3600,
});

await cache.initialize();

// Store with cost tracking
await cache.store('What is the capital of France?', 'Paris', {
  model: 'gpt-4o-mini',
  inputTokens: 20,
  outputTokens: 5,
});

// Exact match - always high confidence
const exact = await cache.check('What is the capital of France?');
// exact.hit === true
// exact.confidence === 'high'
// exact.similarity === 0.0000
// exact.costSaved === 0.0000085

// Paraphrase - typically 'uncertain' at threshold 0.1, 'high' at threshold 0.15
const paraphrase = await cache.check('What city is the capital of France?');
// paraphrase.hit === true
// paraphrase.confidence === 'high'    // at threshold 0.15
// paraphrase.similarity ~= 0.087      // observed with text-embedding-3-small
// paraphrase.costSaved === 0.0000085

This library uses cosine distance (0-2 scale, lower = more similar):

A lookup is a hit when score <= threshold. The default threshold is 0.1.

With real embeddings (text-embedding-3-small):

• Exact same phrasing: ~0.000 - always high
• Close paraphrase ("Which city is the capital of France?"): ~0.08–0.09 - uncertain at default 0.1 threshold, high at 0.15
• Loose paraphrase ("France's capital?"): ~0.10–0.15 - typically miss at 0.1, uncertain at 0.15

Recommended thresholds by use case:

When a hit lands in the uncertainty band (threshold - uncertaintyBand < score <= threshold), you can supply a judgeFn to adjudicate automatically instead of handling confidence: 'uncertain' yourself.

const result = await cache.check(userPrompt, {
  judge: {
    judgeFn: async ({ prompt, response, similarity, threshold, category }) => {
      // Return true to accept (confidence → 'high')
      // Return false to reject (treated as miss with nearestMiss)
      const verdict = await openai.chat.completions.create({
        model: 'gpt-5-mini',
        messages: [
          { role: 'system', content: 'Reply YES or NO only.' },
          { role: 'user', content: `Does this cached response correctly answer the prompt?\nPrompt: ${prompt}\nResponse: ${response}` },
        ],
      });
      return verdict.choices[0].message.content?.startsWith('YES') ?? false;
    },
    onError: 'accept',  // fail-open on judge errors (default)
    timeoutMs: 2000,    // per-call timeout (default)
  },
});

When the judge is invoked: only for confidence === 'uncertain' hits. High-confidence hits, misses, and the zero-candidates case bypass the judge entirely.

Accept path: result.hit === true, result.confidence === 'high'.

Reject path: result.hit === false, result.nearestMiss populated with deltaToThreshold <= 0 (use this to distinguish judge rejections from regular misses where deltaToThreshold > 0).

Composing with rerank: when both rerank and judge are set, the judge receives the reranked pick's response and similarity score.

checkBatch() does not support judge. Call check() individually for prompts that need adjudication.

Store token counts at cache-time to get per-hit cost savings:

await cache.store('What is the capital of France?', 'Paris', {
  model: 'claude-haiku-4-5',   // looked up in bundled LiteLLM price table
  inputTokens: 42,
  outputTokens: 12,
});

const result = await cache.check('Capital of France?');
console.log(result.costSaved);  // e.g. 0.000064 (dollars saved on this hit)

const stats = await cache.stats();
console.log(stats.costSavedMicros); // cumulative microdollars saved

Cost savings scale with the model. Observed values from live examples:

• gpt-4o-mini: ~$0.000006 per hit (cheap model, short responses)
• claude-haiku-4-5: ~$0.000064 per hit (~10x more expensive)
• gpt-4o: ~$0.000100 per hit at 20 input / 5 output tokens

Starting in 0.2.0, initialize() writes a small advisory record to a shared __betterdb:caches hash on the Valkey instance so Monitor (and other tooling) can enumerate caches without configuration. A 60s-TTL heartbeat key is refreshed every 30s; flush() and dispose() remove the heartbeat immediately. No sensitive data is ever written — only cache metadata (type, prefix, version, capabilities, configured thresholds).

Opt out by passing discovery: { enabled: false }. See SemanticCacheOptions.discovery for the full set of knobs.

If your Valkey runs with ACLs, grant the library's user access to the __betterdb:* prefix:

ACL SETUSER <user> +@write +@read ~__betterdb:* ~<your-cache-prefix>:*

Discovery writes are best-effort — if the ACL denies them, the cache still functions and the semantic_cache_discovery_write_failed_total counter increments so operators can alert.

Graceful shutdown: stops the heartbeat and deletes this instance's heartbeat key so Monitor marks the cache offline immediately. Does not drop the index or delete entries. Call from your SIGTERM handler alongside client.quit().

Creates or reconnects to the Valkey search index. Must be called before check() or store(). Safe to call multiple times.

prompt is string | ContentBlock[]. Returns CacheCheckResult:

Options: threshold, category, filter, k, staleAfterModelChange, currentModel, rerank

prompt is string | ContentBlock[]. Returns the Valkey key.

Options: ttl, category, model, metadata, inputTokens, outputTokens, temperature, topP, seed

Stores structured ContentBlock[] as the response. On hit, check() returns contentBlocks.

Pipelined multi-prompt lookups. ~50-70% faster than sequential check() calls. Returns results in input order.

Delete entries matching a valkey-search filter (e.g. '@model:{gpt-4o}').

Convenience wrappers around invalidate().

Returns { hits, misses, total, hitRate, costSavedMicros }.

Returns { name, numDocs, dimension, indexingState }.

Drops the index and all entries. Call initialize() again to rebuild. Also stops the discovery heartbeat and deletes its heartbeat key, but preserves the registry entry in __betterdb:caches so Monitor retains history.

Stops the analytics client, cancels the stats snapshot timer, and disposes the discovery heartbeat. Safe to call multiple times.

Graceful shutdown of the discovery layer for in-process caches without destroying data. Stops the discovery heartbeat and deletes the heartbeat key; does not touch the index or entries.

Analyzes the rolling similarity score window (last 10,000 entries, up to 7 days) and returns:

{
  recommendation: 'tighten_threshold' | 'loosen_threshold' | 'optimal' | 'insufficient_data',
  recommendedThreshold?: number,  // present when recommendation is tighten/loosen
  reasoning: string,              // human-readable explanation
  hitRate: number,
  uncertainHitRate: number,       // >20% triggers tighten recommendation
  nearMissRate: number,           // >30% with avg delta <0.03 triggers loosen
  // ...
}

Returns one result per category seen in the window, plus one aggregate 'all' result.

Every public method emits an OTel span. Requires an OpenTelemetry SDK in the host application.

Runnable examples in examples/. All examples connect to localhost:6399 by default (override via VALKEY_HOST / VALKEY_PORT).

SemanticCache does not own the iovalkey client:

const client = new Valkey({ host: 'localhost', port: 6399 });
const cache = new SemanticCache({ client, embedFn });
// ... use cache ...
await client.quit();

flush() fans out via clusterScan() across all master nodes. FT.SEARCH routes correctly via hash slots. FT.CREATE only creates the index on the receiving node - in a full cluster, create the index on each node separately.

store() requires a complete response string. The Vercel AI SDK adapter does not implement wrapStream. Accumulate the full streamed response before calling store().

v0.2.0 added binary_refs, temperature, top_p, seed fields to the index schema. Existing v0.1.0 indexes operate in text-only mode until flush() + initialize() rebuilds the schema.

MIT