Skip to content

pipeline-manager: restrict edit runtime_config.resources.storage_class#6358

Merged
snkas merged 1 commit into
mainfrom
sc-change
Jun 2, 2026
Merged

pipeline-manager: restrict edit runtime_config.resources.storage_class#6358
snkas merged 1 commit into
mainfrom
sc-change

Conversation

@lalithsuresh

@lalithsuresh lalithsuresh commented May 31, 2026

Copy link
Copy Markdown
Contributor
  • Only allow editing runtime_config.resources.storage_class if the pipeline storage is cleared
  • Align source and test code to the same few restricted fields
  • Integration tests for the edit-restricted fields being edited (enterprise and non-enterprise)
  • Documentation for edit-restricted runtime configuration fields

PR information

  • Unit tests updated
  • Integration tests added
  • Documentation updated
  • Changelog updated
  • One backward incompatible change, but it is a fix for behavior that should be in place: no longer able to edit runtime_config.resources.storage_class if the storage is not cleared.

@lalithsuresh lalithsuresh requested a review from ryzhyk May 31, 2026 16:35
@ryzhyk ryzhyk requested a review from snkas May 31, 2026 17:11

@mythical-fred mythical-fred left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Symmetric with the existing restriction, and the test-model alignment commit (dropping the spurious /// model-only checks) is a nice bonus — model and source now agree on what is restricted, so the proptest harness () actually exercises the new branch via the strategy that varies .

One soft nit, non-blocking: the defaulting trick on the field is load-bearing (older pipelines may lack the key) but it is a little obscure for a future reader — a brief comment would help. The new / checks correctly use -equality so a missing key on both sides compares equal without that workaround. Fine as is.

🐟

@mythical-fred

Copy link
Copy Markdown

My APPROVE review just above arrived with most of its code spans eaten by a shell-substitution bug on my end (unquoted heredoc + backticks — same trap I burned myself on in #6355, lesson clearly not learned). Posting the intended text here.


LGTM. Symmetric with the existing runtime_config.resources.namespace restriction, and the test-model alignment commit (dropping the spurious program_code / udf_rust / udf_toml / program_config model-only checks) is a nice bonus — model and source now agree on what is restricted, so the proptest harness (StorageAction::UpdatePipeline) actually exercises the new branch via the ResourceConfig strategy that varies storage_class.

One soft nit, non-blocking: the let one = json!(1); defaulting trick on the hosts field is load-bearing (older pipelines may lack the key) but it is a bit obscure for a future reader — a brief comment would help. The new storage_class / namespace checks correctly use Option-equality so a missing key on both sides compares equal without that workaround. Fine as is.

🐟

@snkas snkas left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, only thing that would be handy is adding a Python test as it is e2e behavior that can be tested.

Comment thread crates/pipeline-manager/src/db/test.rs

@mythical-fred mythical-fred left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-APPROVE on 834e15e.

Three nice things landed since my last pass:

  1. The runtime_config.hosts comparison in db/operations/pipeline.rs dropped the unwrap_or(&one) defaulting. That was an unrelated latent bug — None vs Some(1) and Some(null) would all collapse together and silently allow a transition between them. Straight equality is correct.
  2. storage_class restriction added symmetrically with namespace, both source and test model.
  3. The test-model ModelHelpers impl now matches source exactly — the spurious model-only checks on program_code/udf_rust/udf_toml/program_config are gone, and the storage field check (which doesn't exist in source) is replaced with the real hosts check. Model and source now agree on what is restricted, which is the whole point of the model.
  4. test_runtime_config_edit_restricted[_enterprise] parameterizes the lifecycle across workers / resources.storage_mb_max / resources.namespace / resources.storage_class (+ hosts / fault_tolerance enterprise-only). Good coverage that addresses snkas's e2e ask from earlier today.

Two non-blocking nits on the python helper:

  • helper_test_restricted_runtime_config_edit ends with TEST_CLIENT.patch_pipeline(name=pipeline_name, runtime_config={}) to reset between fields. That works because the test sequencing happens to leave storage cleared at that point, but it's an implicit ordering dependency between calls. A short comment ("storage is cleared by the preceding successful patch, so this empty patch is allowed") would make the chain less fragile to future edits.
  • The enterprise_only test creates a fresh pipeline rather than reusing the one from test_runtime_config_edit_restricted. Fine, but if you ever wanted to assert that hosts / fault_tolerance are also gated by the storage-cleared check in non-enterprise mode (where they should silently no-op or 400), there's no coverage. Tracking-issue territory, not a blocker.

@snkas snkas changed the title pipeline-manager: prevent changing storage_classes without clearing storage pipeline-manager: restrict edit runtime_config.resources.storage_class Jun 1, 2026
- Only allow editing `runtime_config.resources.storage_class` if the
  pipeline storage is cleared
- Align source and test code to the same few restricted fields
- Integration tests for the edit-restricted fields being edited
  (enterprise and non-enterprise)
- Documentation for edit-restricted runtime configuration fields

Co-authored-by: Lalith Suresh <lalith@feldera.com>
Signed-off-by: Lalith Suresh <lalith@feldera.com>
Signed-off-by: Simon Kassing <simon.kassing@feldera.com>
@snkas snkas added this pull request to the merge queue Jun 1, 2026
@github-merge-queue github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Jun 1, 2026
@snkas snkas added this pull request to the merge queue Jun 2, 2026
Merged via the queue into main with commit 3eb6252 Jun 2, 2026
1 check passed
@snkas snkas deleted the sc-change branch June 2, 2026 09:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants