Tags: diskerror/ragger.cpp
Tags
summarizer: strip_thinking, quality knobs, junk filters, import pipeline
- src/summarizer.cpp: add strip_thinking() to remove <thinking> blocks
and bare 'Thinking Process:' preambles; use rfind for last blank line
- include/ragger/summarizer.h: declare strip_thinking()
- tests/test_summarizer.cpp: 5 tests for strip_thinking including edge cases
- include/ragger/config.h: default cleanup_max_age_hours 336 -> 0 (keep forever)
- docs/configuration.md, example-settings.ini: update comments for new default
- scripts/embedding_comparison.py: remove (raw+1)/2 normalisation — report
raw cosine directly; the mapping assumed bipolar embeddings but MiniLM
produces non-negative vectors, making 0.5 the floor and masking real signal
- scripts/import/: new import pipeline
helpers.py shared config, DB, LLM, dedup, junk filter utilities
import_openclaw.py OpenClaw daily notes + session files; strips metadata
blobs, junk lines, session headers; ts from filename
or embedded '# Session: YYYY-MM-DD HH:MM:SS UTC' header
import_claude.py Claude Code project memory files
import_gemini.py Gemini GEMINI.md bullets
import_hermes.py Hermes MEMORY.md section blocks
import_claude_export.py Claude.ai web export conversations -> turns table
import_all.py runs all scripts in order
feat(summarizer): L3 from L2s, L4 from L3s — layered running summaries L3 now summarizes L2 turn summaries (not raw turns). L4 now summarizes all complete L3 session summaries (was unimplemented). Both are running summaries: a single row is created on first write and updated in-place on every subsequent change, rather than accumulating new rows. Flow: L2 write succeeds → enqueue L3UpdateSession for that session L3UpdateSession → upsert status='current' session row from all L2s pause timer fires → enqueue L3CloseSession for idle sessions L3CloseSession → mark session row complete, enqueue L4UpdateProject L4UpdateProject → upsert status='current' project row from all complete L3s New helpers in sqlite_backend: l2_summary_texts(session_guid) — non-draft L2 texts, oldest-first complete_l3_summary_texts() — complete L3 texts, oldest-first current_project_summary() — the single running L4 row New summarize_texts() in summarizer.cpp — same pct-based size contract as summarize_transcript() but takes a flat list of blobs, appropriate for summarizing summaries rather than user/assistant pairs. sessions_needing_close() SQL tightened: now requires at least one non-draft L2 (avoids closing sessions with only heuristic drafts). All 14 unit tests pass.
fix(turns): dedup retry/regeneration turns (keep-latest) When the agent re-answers the same user prompt without an intervening new prompt, capture_turn fired store_turn a second time and INSERTed a duplicate row — same user_text, a new assistant_text — re-embedding and double-indexing the user side in turns_fts. Observed both as same-session regenerations (32/33, 45/46) and across freshly-minted session GUIDs when the TUI crash-restarted (134/135). store_turn now checks the most-recent turn before inserting: if its user_text is identical and it falls within a 5-minute window, update that row in place (keep-latest assistant_text + re-embed) and return its id instead of inserting. The match is intentionally cross-session — a session change between two identical prompts is itself a breakage signal, not a reason to keep both. The FTS au-trigger keeps the index consistent; empty (partial/prompt-arrival) assistant rows never dedup. This mitigates the duplicate-row symptom only. The real fix — grouping turns into stable threads across frontend restarts — is tracked in #70. Adds test_store_turn_dedup. Full suite 13/13 green.
Token display/rotation, deploy log truncation - add-self: prints token for client config setup - add-user: prints token file location reminder - GET /user/token: returns authenticated user's current token - POST /user/rotate-token: rotates token, updates DB, returns new token - Web UI: /token and /rotate-token chat commands - deploy.sh: truncate logs on deploy
Replace Crow with cpp-httplib — real SSE streaming - Swapped Crow HTTP framework for cpp-httplib (single header) - /chat endpoint now streams SSE tokens in real-time via set_chunked_content_provider() instead of buffering entire response - Background thread feeds tokens through shared queue to DataSink - All 16 routes converted to httplib API - Static file serving via httplib set_mount_point() - Removed Boost.Asio dependency (only Boost.ProgramOptions remains) - TLS: native support deferred (use reverse proxy); was Crow-specific - 12/12 native tests pass
Add SIGHUP config reload + 'ragger reload' CLI command - config.h/cpp: reload_config() re-reads INI, updates Config struct in-place. Restart-required keys logged to stderr but not applied. Uses RELOAD macro for clean field-by-field comparison. - server.cpp: SIGHUP handler sets atomic flag, timer loop checks it. Re-initializes InferenceClient when config changes. - main.cpp: 'ragger reload' finds running daemon via PID file, sends SIGHUP. Added to help text.
PreviousNext