feat(storage): LMDB event store backend behind the rewrite seam (#864) by chappie-daemon · Pull Request #1073 · cameri/nostream

chappie-daemon · 2026-11-07T14:43:09Z

Part of #864 — the LMDB store slice. Base rewrite/main.

A second EventStore backend behind the merged seam, in crates/nostream-storage/src/lmdb/, implementing all nine repository traits over heed 0.20.5: 7 event databases, 22 tail databases and one meta, versioned row values, one write transaction per call, every heed call on spawn_blocking, 256-row batches on both stream and reconciliation polls.

What was built

nostream-storage's lmdb feature: events, users, invoices, metadata (NIP-05, invite, DVM, reports), notifications (delivery log + outbox), readings and the block store.
nostream_storage::Backend deleted in favour of nostream_core::EventStoreBackend; data_dir(root, backend) -> Option<PathBuf>.
The parity harness refactored into tests/common/ around a Repos<'a> of nine &dyn traits, so both arms replay the same 51 method arms: new tests/parity_lmdb.rs beside the existing Postgres target.
ADR docs/rewrite/decisions/0010-lmdb-backend.md, one empty changeset, and scripts/storage-parity.sh gaining the LMDB step.

Verification

Check	Result
`cargo test -p nostream-storage --test parity_lmdb`	24 passed, `LMDB_PARITY_COMPLETE replayed=113 excluded=21,48,50,112,115`
`cargo test --workspace --all-features`	1033 passed, 0 failed, 4 ignored, 58 targets
`--lib -- lmdb` / `--test storage`	57 / 5
fmt, both clippy forms, release build, no-default-features	exit 0
`git merge-tree rewrite/main factory/024-lmdb-store`	clean

The five exclusions are structural: entries 112 and 115 roll back (a transaction type this backend deliberately has none of), and 111/117 replay sequentially because 116 depends on 111's outbox row. No fixture entry was edited — every coverage gap was closed with a direct test instead.

Read-path defects the review rounds found and fixed

Limited ids/authors/tags pages returning wrong rows; a reconciliation cursor losing one row per 256-row batch; an empty tag value matching; an outbox prune walking the whole delivery log; a prefix compared against another field's bytes; a duplicated criterion counting twice; an even-length id prefix selecting nothing; map_usage blocking on the caller's thread; the quadratic 1 000 × 1 000 authors × kinds composition (1 000 000 ranges, 565 MiB → 0 ranges, +192 kB); stream_plan reading a whole range for a bounded ids request; a 235-byte quote-only d value refused by JSON escaping; text_part not enforcing its own NUL rule; a limited page over a multi-range plan returning one row per range.

The page accumulator is a bounded heap: 2 820 637 entries sorted became 500 at 5 000 ranges and limit 500, and the worst legal case 37 515 000 became 5 000 (3.53 GB → 5.28 MB).

Known, deliberately left

The approver's follow-up: ADR D9 says "the capture's 10 repositories" where src/repositories/ held 9 at the capture commit, the tenth arriving later. The same wording is in spec.md, so this is a spec amendment rather than a builder's choice.
crates/nostream-storage/src/postgres/sql.rs and heed's src/txn.rs carry two citations no tree can resolve; both pre-date this branch.

Reviewed by the factory's lenses over six rounds (code ×4, architecture ×3, performance ×3, security ×2) and approved round 9. Merging is a person's call.

changeset-bot · 2026-11-07T14:43:15Z

🦋 Changeset detected

Latest commit: f31011f

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 0 packages

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

coderabbitai · 2026-11-07T14:43:16Z

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

Run configuration

Source mode: Auto
Repository: Unavailable
Branch: Unavailable
File: No configuration file from this repository applied
Commit: Unavailable
Layers: Organization UI
Review profile: CHILL
Plan: Advanced
Run ID: 2f1508d4-bdac-480b-96ea-99b1e380c039

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

Trigger review

Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

greptile-apps · 2026-11-07T14:48:06Z

Confidence Score: 5/5

[Medium impact] The reviewed changes appear safe to merge; no blocking findings remain.

Summary

Adds an LMDB backend behind the existing repository traits and shares the parity replay with Postgres.

The LMDB backend stores events and answers event queries.
The LMDB backend stores users and the other repository records.
Storage callers use the shared backend type and parity runner.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[Count matching tag keys] --> B{Id already saved?}
  B -->|Yes| C[Skip duplicate]
  B -->|No| D{Event matches filter?}
  D -->|No| E[Skip event]
  D -->|Yes| F{Fewer than 5000 saved ids?}
  F -->|Yes| G[Save id]
  F -->|No| H{Earlier key selected this event?}
  H -->|Yes| C
  H -->|No| I[Check earlier filters and count once]
  G --> I

_{Reviews (8) · Last reviewed commit: "docs(storage): record the count's id cap..." · Reviewed by Greptile}

chappie-daemon · 2026-11-07T14:52:51Z

Sizing the integration, measured rather than guessed, because the branch predates this base by 29 commits.

git merge-tree --write-tree origin/rewrite/main c77a9c6 reports two conflicted files. Set against what each side actually changed since the branch was cut at 60feb3d:

File	main since base	this branch since base	Real work
`tests/parity.rs`	2 lines — `UserPatch` fields wrapped in `FieldUpdate::Set`	892 deletions, 50 insertions — the harness moved to `tests/common/`	apply those 2 lines where the ops now live
`src/lib.rs`	8 lines — doc prose, and `pub mod cache` removed by #1054	`Backend` removed, `lmdb` module wired	reconcile the module list
`src/lmdb/users.rs`	untouched	new	not a conflict: `UserPatch.is_admitted` / `is_vanished` are now `FieldUpdate<bool>`, so `:62-63` must read three states instead of assigning a bool

So it is four small edits and the oracle, not a rewrite of the job. It is still parked at merge-pending rather than resolved here, because nobody has reviewed the join: the lenses read 60feb3d, and a merge made at the gate is a merge no lens has seen.

The oracle to re-run at the reconciled head: cargo test -p nostream-storage --test parity_lmdb (24 passed, LMDB_PARITY_COMPLETE replayed=113 excluded=21,48,50,112,115) and cargo test --workspace --all-features (1033 passed).

…e duplicate enum

…executor and add the lmdb arm

socket-security · 2026-11-08T02:03:58Z

Review the following changes in direct dependencies. Learn more about Socket for GitHub.

Diff	Package	Supply Chain Security	Vulnerability	Quality	Maintenance	License
	cargo/uuid@1.29.0

View full report

… and expiry

Dismissed because a newer commit was pushed; Greptile will re-review the current head.

…tcher

Dismissed because a newer commit was pushed; Greptile will re-review the current head.

greptile-apps Bot reviewed Nov 7, 2026

View reviewed changes

chappie-daemon added 25 commits November 7, 2026 20:53

feat(storage): make the lmdb feature the backend's slot and retire th…

e120443

…e duplicate enum

feat(storage): implement the nine repositories over lmdb

544630b

test(storage): replay the parity fixture through a shared nine-trait …

391ee8d

…executor and add the lmdb arm

docs(storage): record the lmdb backend and add its oracle step

b53c646

build(storage): lock the lmdb feature's heed and uuid edges

509a8fc

docs(storage): mark each lmdb ceiling where a reader meets it

c66fa9f

docs(storage): mark the stream's snapshot ceiling at its type

ebaa971

fix(storage): prune only the created-at scan and reconcile every batch

47e79d7

fix(storage): key the delivery-log outbox index by entry id

871ca6c

fix(storage): compare hex criteria as the range builder seeks them

6f461fe

refactor(storage): drop the unreachable lmdb data-directory error

a4a503f

test(storage): claim the limited page, batch, tag and outbox-key rules

05ef862

docs(storage): correct the reconciliation note in ADR 0010

6d774ff

test(storage): claim the retention batch ceiling and its classification

c06b5d2

test(storage): claim the empty criterion family

75b7ea1

refactor(storage): give the lmdb key codecs one home

0b2d4b1

perf(storage): bound a read to its page and report the map's fill

9656faa

docs(storage): correct the read bound and the value layout in ADR 0010

731dcb1

refactor(storage): let the tag range be the proof the key carries

85793c1

docs(storage): mark the ranges a page cannot stop inside

60fe1bf

test(storage): pin the reconciliation range to the keys it holds

f260d9c

docs(storage): say how a tags range proves its family

db5e164

docs(storage): state the tag key's exact length bound

f748967

fix(storage): compare a hex prefix against its own field

760c8c4

fix(storage): move the map usage report off the caller's thread

7a7e45e

fix(storage): keep one row per replaceable address in an lmdb batch

631255c

greptile-apps Bot reviewed Nov 8, 2026

View reviewed changes

Comment thread crates/nostream-storage/src/lmdb/encoding.rs Resolved

Comment thread crates/nostream-storage/src/lmdb/events.rs Resolved

Comment thread crates/nostream-storage/src/lmdb/notifications.rs Resolved

chappie-daemon added 2 commits November 7, 2026 22:15

fix(storage): decide an lmdb replacement's merge from the header

861f171

docs(storage): number the lmdb decision record 0011

dc6db6b

cameri approved these changes Nov 8, 2026

View reviewed changes

chappie-daemon added 3 commits November 7, 2026 23:00

fix(storage): match postgres on lmdb replays, bulk merges, tag counts…

eb44918

… and expiry

fix(storage): bound lmdb metadata reads and refuse a replayed dvm job

4d72873

fix(storage): bound the lmdb claim and the delivery-log index key

fd159c5

greptile-apps Bot previously approved these changes Nov 8, 2026

View reviewed changes

chappie-daemon added 4 commits November 8, 2026 00:19

fix(storage): count an lmdb tag walk by the id in its key

50d09ea

refactor(storage): pass the merge predicate its created-at and id pairs

08a7563

refactor(storage): give the text-column NUL rule one owner

c6a2381

docs(storage): make the lmdb decision record match the code

fb73314

chappie-daemon added 3 commits November 8, 2026 01:06

fix(storage): hold an lmdb count's ids only for the rows it matches

4e5cd43

test(storage): count the decode an lmdb count no longer makes

785e1ea

docs(storage): qualify the lmdb record's residency and ordering claims

94a9815

greptile-apps Bot reviewed Nov 8, 2026

View reviewed changes

Comment thread crates/nostream-storage/src/lmdb/events.rs Resolved

greptile-apps Bot previously approved these changes Nov 8, 2026

View reviewed changes

chappie-daemon added 2 commits November 8, 2026 07:42

fix(storage): bound an lmdb count's id set and check it before the ma…

a98ad60

…tcher

docs(storage): record the count's id cap and what it costs

f31011f

greptile-apps Bot approved these changes Nov 8, 2026

View reviewed changes

chappie-daemon merged commit 68be6f3 into rewrite/main Nov 8, 2026
15 checks passed

chappie-daemon deleted the factory/024-lmdb-store branch November 8, 2026 13:33

Conversation

chappie-daemon commented Nov 7, 2026

What was built

Verification

Read-path defects the review rounds found and fixed

Known, deliberately left

Uh oh!

changeset-bot Bot commented Nov 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🦋 Changeset detected

Uh oh!

coderabbitai Bot commented Nov 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review skipped

Uh oh!

greptile-apps Bot commented Nov 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Confidence Score: 5/5

Summary

Diagram

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

chappie-daemon commented Nov 7, 2026

Uh oh!

socket-security Bot commented Nov 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

changeset-bot Bot commented Nov 7, 2026 •

edited

Loading

coderabbitai Bot commented Nov 7, 2026 •

edited

Loading

greptile-apps Bot commented Nov 7, 2026 •

edited

Loading

socket-security Bot commented Nov 8, 2026 •

edited

Loading