Skip to content

feat(cli)!: remove legacy cloud sync and bisync - #1953

Merged
phernandez merged 1 commit into
mainfrom
feat/1845-remove-cloud-mirror-sync
Oct 29, 2026
Merged

phernandez merged 1 commit into
mainfrom
feat/1845-remove-cloud-mirror-sync

Conversation

@phernandez

Copy link
Copy Markdown
Member

Why

bm cloud sync made Cloud match the local folder with rclone sync --delete-excluded. It deleted cloud-only notes, including notes that people and managed agents created in Cloud. bm cloud bisync mirrored both ways and could delete or overwrite files on either side. #1596 deprecated both. For 1.0, bm cloud push / bm cloud pull replace them on Personal and Team workspaces. They never delete on the destination and stop on differing files until the user chooses a conflict strategy.

What changed

Removed

  • project_sync, project_bisync, get_project_bisync_state, bisync_initialized, get_rclone_version, supports_create_empty_src_dirs, and MIN_RCLONE_VERSION_EMPTY_DIRS from rclone_commands.py.
  • The sync / bisync / bisync-reset implementations, MirrorCommand, _warn_mirror_deprecated, and the Team-refusal messages for the mirrors from project_sync.py.
  • The force / fail_on_read_error options of the .bmignore converter. Only the destructive sync used them. The test file that covered them is deleted.
  • bm project remove no longer deletes bisync-state/<project>. Nothing reads that state now, and the migration contract says old state directories are left for the user.

Kept as migration stubs (cloud/removed_mirror_commands.py)

  • sync, bisync, and bisync-reset are hidden from bm cloud --help. They accept any of the old arguments and print pull/push commands for the named project (shell-quoted, with <project> when no name was given). The output also covers the deletion, rename, and conflict differences. Each stub then exits 1.
  • The stubs read no config or credentials, call no API, and touch no files. They do not forward to push or pull.

Kept and moved (the old bisync_commands.py holds no bisync code now)

  • mount_api.py: get_mount_info (used by push/pull, prune, check, and project ls --cloud) and generate_mount_credentials (used by cloud setup). BisyncError is renamed MountApiError.
  • rclone_filters.py: convert_bmignore_to_rclone_filters (used by the push/pull diff and copy, and by check) and convert_bmignore_to_rclone_prune_filters (used by prune). rclone_commands.py now imports them at the top; the earlier lazy import was not needed to avoid a cycle.
  • _require_personal_workspace now serves only prune, so its message parameter is gone.

Unchanged: push/pull (dry-run, --workspace routing, --on-conflict choices, destination-only preservation), Team WebDAV transfer, cloud setup, sync-setup, prune, and check. The bisync_initialized / last_sync config fields stay so existing config files still load.

Docs: docs/1.0.md (status, stub behavior, state-directory note, checklist), docs/cloud-cli.md (the deprecated section is replaced by "Removed in 1.0" with a migration table; the limitations now cover renames and reverse-transfer resurrection), bm(1), AGENTS.md, and the Hermes README.

Verification

  • just fast-check: lint, format, and ty pass. (This worktree needed uv sync --extra milvus first so ty could resolve pymilvus.)
  • Issue baseline: BASIC_MEMORY_TESTMON_SELECT_FLAGS="--import-mode=importlib" just fast-test tests/cli/cloud/test_project_sync_command.py tests/cli/cloud/test_webdav_transfer.py tests/test_rclone_commands.py passes 136 tests. The original baseline was 194; the difference is the deleted mirror tests.
  • Targeted run: tests/cli/cloud, tests/test_rclone_commands.py, tests/cli/test_project_remove_cloud_stub.py, tests/test_remedy_emission_sites.py, tests/test_config.py, tests/cli/test_project_set_cloud_local.py, tests/cli/test_project_add_with_local_path.py, tests/cli/test_project_list_and_ls.py, tests/test_man_pages.py, and tests/cli/test_man_command.py pass 529 tests.
  • All of tests/cli: 1262 passed, 7 skipped.
  • New tests/cli/cloud/test_removed_mirror_commands.py runs the real CLI against a logged-in config with a synced project and old bisync state. It asserts exit 1, guidance text, and that no user file, config, or bisync state changes. It also blocks every httpx request.

Note: test_project_add_remedy_matrix.py's cloud rows can fail depending on test order when run in one session after the cloud test files. I reproduced the same 4 failures on unmodified main, so this PR did not cause them. The file passes when run alone and in the full tests/cli run.

Fixes #1845

πŸ€– Generated with Claude Code

https://claude.ai/code/session_01CMYVL1x91B8kJxy7nn2GU8

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 29, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
πŸ“ Code Review βœ… Completed 2026-10-29T06:15:16.973395Z 82ddc0a Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with πŸ‘€ while any review is running, comments if it has suggestions, and reacts with πŸ‘ once all reviews finish with no findings.

`bm cloud sync` made Cloud match the local folder with `rclone sync
--delete-excluded`, so it deleted cloud-only notes, including notes that
people and managed agents created in Cloud. `bisync` mirrored both ways
and could delete or overwrite on either side. Both were deprecated in
never delete on the destination and stop on differing files until the
user picks a conflict strategy.

Remove the mirror implementations (project_sync, project_bisync, bisync
state helpers, rclone version probing for --create-empty-src-dirs, and
the forced/fail-fast filter refresh only the destructive sync used).
Keep `sync`, `bisync`, and `bisync-reset` as hidden migration stubs. They
accept the old arguments, print push/pull commands for the named project
and the deletion, rename, and conflict differences, and exit 1. They
read no config or credentials, call no API, and touch no files.

Move the helpers push/pull, prune, check, and setup still use out of
bisync_commands.py: the mount API calls go to mount_api.py
(BisyncError becomes MountApiError) and the .bmignore converters go to
rclone_filters.py. `bm project remove` no longer deletes old bisync state,
which nothing reads now. Update CLI help, bm(1), docs/cloud-cli.md, the
1.0 migration guide, AGENTS.md, and the Hermes README.

Fixes #1845

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CMYVL1x91B8kJxy7nn2GU8
Signed-off-by: phernandez <paul@basicmachines.co>
@phernandez
phernandez force-pushed the feat/1845-remove-cloud-mirror-sync branch from f10ad2d to 82ddc0a Compare October 29, 2026 06:13
@phernandez

Copy link
Copy Markdown
Member Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Another round soon, please!

Reviewed commit: 82ddc0a0d8

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with πŸ‘.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

@phernandez
phernandez merged commit 4f7aa87 into main Oct 29, 2026
58 checks passed
@phernandez
phernandez deleted the feat/1845-remove-cloud-mirror-sync branch October 29, 2026 06:37
phernandez added a commit that referenced this pull request Oct 29, 2026
Refs #1955. The teardown hang is still open there; see the last section.

This PR fixes the flakes reported in #1955 plus two the coordinator
added: the linear-scaling tokenizer tests and the MCP container leak.
Each section gives the cause, the evidence and the change. Verification
is at the end.

## 1. `vtable constructor failed: search_index` in the watcher lifecycle
test (test bug, real SQLite behavior)

**Cause.** The SQLite `engine_factory` fixture in `test-int/conftest.py`
created `search_index` but not the vector storage. So the first
embedding of the new note ran `_ensure_vector_tables`, which created
`search_vector_chunks`, its indexes and the `vec0` table while
`bm_search` polled on another pooled connection. A schema change makes
other connections reload the schema and reconnect the FTS5 table. A
reconnect that overlaps the DDL fails with this error.

**Evidence.**
- CI logs for jobs 123581698132 (main, ab6a059) and 123623036006
(#1962) both fail in the `narwhal` loop with this error.
- I reproduced it locally on main with a DDL tracer, a pytest plugin
that logs every DDL and search statement with timestamps. The failing
`SELECT COUNT(*) FROM search_index` is interleaved with `CREATE INDEX
... search_vector_chunks`, issued from `_ensure_vector_tables` <-
`sync_entity_vectors_internal`, about 1 ms earlier.
- A standalone sqlite3 script gives the same `vtable constructor failed:
search_index`. It commits schema changes from a second connection while
the first prepares an FTS5 query.

**Not a product bug.** Production opens the database through
`get_or_create_db` -> `run_migrations` -> `init_search_index`, which
creates `search_index`, `search_vector_chunks` and
`search_vector_embeddings` before any request. I confirmed this on a
fresh database. The `_ensure_vector_tables` DDL that runs per repository
instance afterwards uses `IF NOT EXISTS`. Re-running it leaves `PRAGMA
schema_version` unchanged (10 -> 10), so it cannot trigger the
reconnect.

**Fix.** The SQLite fixture now calls
`create_search_repository(...).init_search_index()`, as the Postgres
branch already did. After the fix, the trace shows no schema-changing
DDL during the search loops.

The `TimeoutError` symptom (job 123284407567, #1953) times out in the
same `narwhal` loop, right where the first embedding creates the vector
tables. A watcher batch that fails is logged and dropped until the next
change, so the same race on the indexer's write would leave `narwhal`
unindexed. CI lost the loguru output ("I/O operation on closed file"),
so this link is inferred, not shown.

## 2. Module-global `_initial_index_tasks` (product lifecycle fix)

**Cause.** Startup project indexes were fire-and-forget tasks held in a
module-level set. Nothing owned them:
- `WatchCoordinator.stop()` cancelled only the watcher.
- The lifespan could close the database while they still ran.
- The set outlived event loops. A `_CapturedTask` fake from
`tests/services/test_initialization.py` was later gathered by the
lifecycle test (`TypeError`).

**Fix.** `initialize_file_indexing` runs the startup indexes and
`watch_service.run()` in one `asyncio.TaskGroup`. Cancelling the watcher
now cancels and awaits the indexes. The global and the test's
`gather(*_initial_index_tasks)` are gone.

Each index task still catches and logs its own `Exception`, so one
failing project does not cancel the watcher. `CancelledError` still
propagates. A new test,
`test_a_failed_startup_index_leaves_the_watcher_running`, uses a project
whose root is a regular file, so its real index fails, and asserts the
watcher keeps running. The test fails if the handler re-raises.

The teardown hang (loop close stuck in `_cancel_all_tasks`) reproduced
once locally in about 40 loaded runs, with the same stack as CI job
123279312418. It did not reproduce again in 25 more runs with a
pending-task dump installed, so I cannot name the task that ignored
cancellation. This change removes the startup indexes as unowned tasks
that outlived the lifespan.

## 3. `test_file_sqlite_concurrent_writes_commit_and_dispose` (test
design)

**Cause.** The failure is on Windows, main 5b1d843, run 40839542939,
job 123332360458. It was not in the ab6a059 run. SQLite's busy handler
polls with growing sleeps of up to 100 ms instead of queueing waiters.
When 64 writers each ran 40 transactions back to back, the other pooled
connections could pass the write lock between themselves for the whole
run while one waiter slept through every free moment. The run took 19.97
s on Windows against about 3 s on Linux, and one waiter passed
`busy_timeout` (10 s): `database is locked` on the first `INSERT` of a
transaction.

**Fix.** Same 2560 writes, run in 40 rounds of 64 concurrent writers. A
waiting connection now competes only with one round's work, well under a
second, never with the rest of the run. Contention within each round is
unchanged.

**Note, not changed here.** On Windows, `connect_args={"timeout": 30.0}`
is overridden by the per-connection `PRAGMA busy_timeout=10000`, so the
30 s setting never takes effect.

## 4. Postgres refresh negative controls (deleted)

**Cause.** These controls assert that overlapping refreshes produce
`search_index_pkey`. The upsert's `ON CONFLICT` absorbs any conflict it
sees in its arbiter index; it waits for the other transaction, then
updates. The pkey error happens only when two inserts pass the arbiter
check in the same instant. The unit engine uses `NullPool`, so on a
loaded runner one refresh could commit before the last one had deleted
anything, and the round did not collide (job 123284407396).

**Why deleted.** I first replaced the sleep with an `asyncio.Barrier`
between delete and insert. With every delete done before any insert, the
saves control still missed 1 run in 5. Nothing outside the INSERT can
force that instant. The tolerance is already pinned deterministically:
- `tests/services/test_search_refresh_atomicity.py`: an injected lost
race is logged and the previous projection kept; any other database
error still raises.
- `tests/repository/test_search_refresh_race.py`: which errors count as
a lost race.

The positive concurrency tests still assert that overlapping refreshes
never fail.

## 5. Linear-scaling tokenizer tests (robust measurement)

**Cause.** The tests compared wall-clock time (`perf_counter`), which
includes time the test spends descheduled. On a loaded machine, the old
version failed 8 of 10 runs with 24 CPU burners.

**Fix.** The tests now measure `time.thread_time()` (this thread's CPU
only) with `gc` paused, best of 5, and keep the Γ—2.5 bound. There is no
operation count to assert instead: the quadratic versions make the same
calls and inspect the same characters, and only the bytes each slice
copies grow, inside C.
- Under the same load the new version passed 20 of 20 runs.
- The quadratic implementations still fail it (Γ—2.96 and Γ—3.3), checked
by temporarily reverting them.

## 6. MCP container leak (product fix)

**Cause.** The lifespan called `set_container()` and never reset it.
After an in-process `Client(mcp)` session ended, later code in the same
process, such as picking the default search mode, read that session's
config.

**Fix.** A new `basic_memory.mcp.container.installed_container` installs
the container for the lifespan only and restores the previous one on
exit, mirroring the API's `installed_container`. Regression test:
`test_mcp_lifespan_uninstalls_its_container_on_exit`.

## Verification
- `just fast-check`: clean.
- `just test-int-sqlite`, full suite: 720 passed, 14 skipped.
- `just test-int-postgres -k "watch_lifecycle or lifespan or
mcp_read_cache or sqlite_lifecycle"`: passed.
- `just test-unit-sqlite` and `just test-unit-postgres` on the touched
modules: passed.
- Coverage of the touched src: `mcp/container.py` 100%,
`services/initialization.py` 100%. `mcp/server.py` misses only lines
this PR did not change.
- Repro loops: the lifecycle and SQLite contention tests passed 14 of 15
runs under 8 burners, with no vtable errors; the one failure was the
teardown hang described in section 2. Then 25 of 25 under 2 burners. On
main, 1 of about 30 traced or loaded runs hit the vtable error.

πŸ€– Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01CMYVL1x91B8kJxy7nn2GU8

---------

Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
phernandez added a commit that referenced this pull request Oct 29, 2026
## Why

`just release v1.0.0` pre-flight-checks for a `## v1.0.0` heading on
`main`, so the changelog has to land first.

## What changed

`CHANGELOG.md`: the Unreleased section becomes `## v1.0.0 (2026-10-29)`.
Its existing items are kept, and the 60 or so commits since #1871 are
added:

- **Breaking:**
- the MCP surface is only `bm_*`, with the old tools deleted (#1944,
#1947, #1959, #1962–#1964)
  - `project` is required (#1934)
  - the ChatGPT adapters moved to Cloud
  - `bm tool` mirrors the tools (#1943)
  - the POSIX verbs are rebuilt and `bm tail` is removed (#1960)
  - cloud sync/bisync are removed (#1953)
  - `bm schema` project resolution (#1959)
  - LiteLLM is an optional extra
- **Features:** text-file line ranges in `bm_read`; chat imports return
their file paths.
- **Bug fixes:** including the three found in the 1.0 live test (#1971,
#1972, #1974), with contributor credits for @Flynn-Kawolsky (#1957),
@SWSAmor (#1951) and @MbappeWU (#1948).
- **Security:** the CodeRabbit deep-scan hardening batch.

I checked the entries against PR bodies rather than titles, and
corrected #1954, #1948 and #1967 to describe what they actually fixed.

The date is today; change it if the tag lands tomorrow.

πŸ€– Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01CMYVL1x91B8kJxy7nn2GU8

Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(cli)!: remove legacy cloud sync and bisync in v1.0

1 participant