Repository navigation
feat(cli)!: remove legacy cloud sync and bisync - #1953
Merged
Merged
Conversation
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
βΉοΈ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with π while any review is running, comments if it has suggestions, and reacts with π once all reviews finish with no findings. |
`bm cloud sync` made Cloud match the local folder with `rclone sync --delete-excluded`, so it deleted cloud-only notes, including notes that people and managed agents created in Cloud. `bisync` mirrored both ways and could delete or overwrite on either side. Both were deprecated in never delete on the destination and stop on differing files until the user picks a conflict strategy. Remove the mirror implementations (project_sync, project_bisync, bisync state helpers, rclone version probing for --create-empty-src-dirs, and the forced/fail-fast filter refresh only the destructive sync used). Keep `sync`, `bisync`, and `bisync-reset` as hidden migration stubs. They accept the old arguments, print push/pull commands for the named project and the deletion, rename, and conflict differences, and exit 1. They read no config or credentials, call no API, and touch no files. Move the helpers push/pull, prune, check, and setup still use out of bisync_commands.py: the mount API calls go to mount_api.py (BisyncError becomes MountApiError) and the .bmignore converters go to rclone_filters.py. `bm project remove` no longer deletes old bisync state, which nothing reads now. Update CLI help, bm(1), docs/cloud-cli.md, the 1.0 migration guide, AGENTS.md, and the Hermes README. Fixes #1845 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CMYVL1x91B8kJxy7nn2GU8 Signed-off-by: phernandez <paul@basicmachines.co>
phernandez
force-pushed
the
feat/1845-remove-cloud-mirror-sync
branch
from
October 29, 2026 06:13
f10ad2d to
82ddc0a
Compare
Member
Author
|
@codex review |
|
Codex Review: Didn't find any major issues. Another round soon, please! Reviewed commit: βΉοΈ About Codex in GitHubCodex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with π. When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback". |
This was referenced Oct 29, 2026
phernandez
added a commit
that referenced
this pull request
Oct 29, 2026
Refs #1955. The teardown hang is still open there; see the last section. This PR fixes the flakes reported in #1955 plus two the coordinator added: the linear-scaling tokenizer tests and the MCP container leak. Each section gives the cause, the evidence and the change. Verification is at the end. ## 1. `vtable constructor failed: search_index` in the watcher lifecycle test (test bug, real SQLite behavior) **Cause.** The SQLite `engine_factory` fixture in `test-int/conftest.py` created `search_index` but not the vector storage. So the first embedding of the new note ran `_ensure_vector_tables`, which created `search_vector_chunks`, its indexes and the `vec0` table while `bm_search` polled on another pooled connection. A schema change makes other connections reload the schema and reconnect the FTS5 table. A reconnect that overlaps the DDL fails with this error. **Evidence.** - CI logs for jobs 123581698132 (main, ab6a059) and 123623036006 (#1962) both fail in the `narwhal` loop with this error. - I reproduced it locally on main with a DDL tracer, a pytest plugin that logs every DDL and search statement with timestamps. The failing `SELECT COUNT(*) FROM search_index` is interleaved with `CREATE INDEX ... search_vector_chunks`, issued from `_ensure_vector_tables` <- `sync_entity_vectors_internal`, about 1 ms earlier. - A standalone sqlite3 script gives the same `vtable constructor failed: search_index`. It commits schema changes from a second connection while the first prepares an FTS5 query. **Not a product bug.** Production opens the database through `get_or_create_db` -> `run_migrations` -> `init_search_index`, which creates `search_index`, `search_vector_chunks` and `search_vector_embeddings` before any request. I confirmed this on a fresh database. The `_ensure_vector_tables` DDL that runs per repository instance afterwards uses `IF NOT EXISTS`. Re-running it leaves `PRAGMA schema_version` unchanged (10 -> 10), so it cannot trigger the reconnect. **Fix.** The SQLite fixture now calls `create_search_repository(...).init_search_index()`, as the Postgres branch already did. After the fix, the trace shows no schema-changing DDL during the search loops. The `TimeoutError` symptom (job 123284407567, #1953) times out in the same `narwhal` loop, right where the first embedding creates the vector tables. A watcher batch that fails is logged and dropped until the next change, so the same race on the indexer's write would leave `narwhal` unindexed. CI lost the loguru output ("I/O operation on closed file"), so this link is inferred, not shown. ## 2. Module-global `_initial_index_tasks` (product lifecycle fix) **Cause.** Startup project indexes were fire-and-forget tasks held in a module-level set. Nothing owned them: - `WatchCoordinator.stop()` cancelled only the watcher. - The lifespan could close the database while they still ran. - The set outlived event loops. A `_CapturedTask` fake from `tests/services/test_initialization.py` was later gathered by the lifecycle test (`TypeError`). **Fix.** `initialize_file_indexing` runs the startup indexes and `watch_service.run()` in one `asyncio.TaskGroup`. Cancelling the watcher now cancels and awaits the indexes. The global and the test's `gather(*_initial_index_tasks)` are gone. Each index task still catches and logs its own `Exception`, so one failing project does not cancel the watcher. `CancelledError` still propagates. A new test, `test_a_failed_startup_index_leaves_the_watcher_running`, uses a project whose root is a regular file, so its real index fails, and asserts the watcher keeps running. The test fails if the handler re-raises. The teardown hang (loop close stuck in `_cancel_all_tasks`) reproduced once locally in about 40 loaded runs, with the same stack as CI job 123279312418. It did not reproduce again in 25 more runs with a pending-task dump installed, so I cannot name the task that ignored cancellation. This change removes the startup indexes as unowned tasks that outlived the lifespan. ## 3. `test_file_sqlite_concurrent_writes_commit_and_dispose` (test design) **Cause.** The failure is on Windows, main 5b1d843, run 40839542939, job 123332360458. It was not in the ab6a059 run. SQLite's busy handler polls with growing sleeps of up to 100 ms instead of queueing waiters. When 64 writers each ran 40 transactions back to back, the other pooled connections could pass the write lock between themselves for the whole run while one waiter slept through every free moment. The run took 19.97 s on Windows against about 3 s on Linux, and one waiter passed `busy_timeout` (10 s): `database is locked` on the first `INSERT` of a transaction. **Fix.** Same 2560 writes, run in 40 rounds of 64 concurrent writers. A waiting connection now competes only with one round's work, well under a second, never with the rest of the run. Contention within each round is unchanged. **Note, not changed here.** On Windows, `connect_args={"timeout": 30.0}` is overridden by the per-connection `PRAGMA busy_timeout=10000`, so the 30 s setting never takes effect. ## 4. Postgres refresh negative controls (deleted) **Cause.** These controls assert that overlapping refreshes produce `search_index_pkey`. The upsert's `ON CONFLICT` absorbs any conflict it sees in its arbiter index; it waits for the other transaction, then updates. The pkey error happens only when two inserts pass the arbiter check in the same instant. The unit engine uses `NullPool`, so on a loaded runner one refresh could commit before the last one had deleted anything, and the round did not collide (job 123284407396). **Why deleted.** I first replaced the sleep with an `asyncio.Barrier` between delete and insert. With every delete done before any insert, the saves control still missed 1 run in 5. Nothing outside the INSERT can force that instant. The tolerance is already pinned deterministically: - `tests/services/test_search_refresh_atomicity.py`: an injected lost race is logged and the previous projection kept; any other database error still raises. - `tests/repository/test_search_refresh_race.py`: which errors count as a lost race. The positive concurrency tests still assert that overlapping refreshes never fail. ## 5. Linear-scaling tokenizer tests (robust measurement) **Cause.** The tests compared wall-clock time (`perf_counter`), which includes time the test spends descheduled. On a loaded machine, the old version failed 8 of 10 runs with 24 CPU burners. **Fix.** The tests now measure `time.thread_time()` (this thread's CPU only) with `gc` paused, best of 5, and keep the Γ2.5 bound. There is no operation count to assert instead: the quadratic versions make the same calls and inspect the same characters, and only the bytes each slice copies grow, inside C. - Under the same load the new version passed 20 of 20 runs. - The quadratic implementations still fail it (Γ2.96 and Γ3.3), checked by temporarily reverting them. ## 6. MCP container leak (product fix) **Cause.** The lifespan called `set_container()` and never reset it. After an in-process `Client(mcp)` session ended, later code in the same process, such as picking the default search mode, read that session's config. **Fix.** A new `basic_memory.mcp.container.installed_container` installs the container for the lifespan only and restores the previous one on exit, mirroring the API's `installed_container`. Regression test: `test_mcp_lifespan_uninstalls_its_container_on_exit`. ## Verification - `just fast-check`: clean. - `just test-int-sqlite`, full suite: 720 passed, 14 skipped. - `just test-int-postgres -k "watch_lifecycle or lifespan or mcp_read_cache or sqlite_lifecycle"`: passed. - `just test-unit-sqlite` and `just test-unit-postgres` on the touched modules: passed. - Coverage of the touched src: `mcp/container.py` 100%, `services/initialization.py` 100%. `mcp/server.py` misses only lines this PR did not change. - Repro loops: the lifecycle and SQLite contention tests passed 14 of 15 runs under 8 burners, with no vtable errors; the one failure was the teardown hang described in section 2. Then 25 of 25 under 2 burners. On main, 1 of about 30 traced or loaded runs hit the vtable error. π€ Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01CMYVL1x91B8kJxy7nn2GU8 --------- Signed-off-by: phernandez <paul@basicmachines.co> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
phernandez
added a commit
that referenced
this pull request
Oct 29, 2026
## Why `just release v1.0.0` pre-flight-checks for a `## v1.0.0` heading on `main`, so the changelog has to land first. ## What changed `CHANGELOG.md`: the Unreleased section becomes `## v1.0.0 (2026-10-29)`. Its existing items are kept, and the 60 or so commits since #1871 are added: - **Breaking:** - the MCP surface is only `bm_*`, with the old tools deleted (#1944, #1947, #1959, #1962β#1964) - `project` is required (#1934) - the ChatGPT adapters moved to Cloud - `bm tool` mirrors the tools (#1943) - the POSIX verbs are rebuilt and `bm tail` is removed (#1960) - cloud sync/bisync are removed (#1953) - `bm schema` project resolution (#1959) - LiteLLM is an optional extra - **Features:** text-file line ranges in `bm_read`; chat imports return their file paths. - **Bug fixes:** including the three found in the 1.0 live test (#1971, #1972, #1974), with contributor credits for @Flynn-Kawolsky (#1957), @SWSAmor (#1951) and @MbappeWU (#1948). - **Security:** the CodeRabbit deep-scan hardening batch. I checked the entries against PR bodies rather than titles, and corrected #1954, #1948 and #1967 to describe what they actually fixed. The date is today; change it if the tag lands tomorrow. π€ Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01CMYVL1x91B8kJxy7nn2GU8 Signed-off-by: phernandez <paul@basicmachines.co> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
bm cloud syncmade Cloud match the local folder withrclone sync --delete-excluded. It deleted cloud-only notes, including notes that people and managed agents created in Cloud.bm cloud bisyncmirrored both ways and could delete or overwrite files on either side. #1596 deprecated both. For 1.0,bm cloud push/bm cloud pullreplace them on Personal and Team workspaces. They never delete on the destination and stop on differing files until the user chooses a conflict strategy.What changed
Removed
project_sync,project_bisync,get_project_bisync_state,bisync_initialized,get_rclone_version,supports_create_empty_src_dirs, andMIN_RCLONE_VERSION_EMPTY_DIRSfromrclone_commands.py.sync/bisync/bisync-resetimplementations,MirrorCommand,_warn_mirror_deprecated, and the Team-refusal messages for the mirrors fromproject_sync.py.force/fail_on_read_erroroptions of the.bmignoreconverter. Only the destructive sync used them. The test file that covered them is deleted.bm project removeno longer deletesbisync-state/<project>. Nothing reads that state now, and the migration contract says old state directories are left for the user.Kept as migration stubs (
cloud/removed_mirror_commands.py)sync,bisync, andbisync-resetare hidden frombm cloud --help. They accept any of the old arguments and printpull/pushcommands for the named project (shell-quoted, with<project>when no name was given). The output also covers the deletion, rename, and conflict differences. Each stub then exits 1.Kept and moved (the old
bisync_commands.pyholds no bisync code now)mount_api.py:get_mount_info(used by push/pull, prune, check, andproject ls --cloud) andgenerate_mount_credentials(used bycloud setup).BisyncErroris renamedMountApiError.rclone_filters.py:convert_bmignore_to_rclone_filters(used by the push/pull diff and copy, and by check) andconvert_bmignore_to_rclone_prune_filters(used by prune).rclone_commands.pynow imports them at the top; the earlier lazy import was not needed to avoid a cycle._require_personal_workspacenow serves onlyprune, so its message parameter is gone.Unchanged: push/pull (dry-run,
--workspacerouting,--on-conflictchoices, destination-only preservation), Team WebDAV transfer,cloud setup,sync-setup,prune, andcheck. Thebisync_initialized/last_syncconfig fields stay so existing config files still load.Docs:
docs/1.0.md(status, stub behavior, state-directory note, checklist),docs/cloud-cli.md(the deprecated section is replaced by "Removed in 1.0" with a migration table; the limitations now cover renames and reverse-transfer resurrection),bm(1), AGENTS.md, and the Hermes README.Verification
just fast-check: lint, format, and ty pass. (This worktree neededuv sync --extra milvusfirst so ty could resolvepymilvus.)BASIC_MEMORY_TESTMON_SELECT_FLAGS="--import-mode=importlib" just fast-test tests/cli/cloud/test_project_sync_command.py tests/cli/cloud/test_webdav_transfer.py tests/test_rclone_commands.pypasses 136 tests. The original baseline was 194; the difference is the deleted mirror tests.tests/cli/cloud,tests/test_rclone_commands.py,tests/cli/test_project_remove_cloud_stub.py,tests/test_remedy_emission_sites.py,tests/test_config.py,tests/cli/test_project_set_cloud_local.py,tests/cli/test_project_add_with_local_path.py,tests/cli/test_project_list_and_ls.py,tests/test_man_pages.py, andtests/cli/test_man_command.pypass 529 tests.tests/cli: 1262 passed, 7 skipped.tests/cli/cloud/test_removed_mirror_commands.pyruns the real CLI against a logged-in config with a synced project and old bisync state. It asserts exit 1, guidance text, and that no user file, config, or bisync state changes. It also blocks every httpx request.Note:
test_project_add_remedy_matrix.py's cloud rows can fail depending on test order when run in one session after the cloud test files. I reproduced the same 4 failures on unmodifiedmain, so this PR did not cause them. The file passes when run alone and in the fulltests/clirun.Fixes #1845
π€ Generated with Claude Code
https://claude.ai/code/session_01CMYVL1x91B8kJxy7nn2GU8