[types] Revert change of ConnectorConfig::max_queued_records to `Op…#6428
Merged
Conversation
ryzhyk
approved these changes
Jun 9, 2026
…tion`. Commit 753cdd8 ("Add support for bytewise limiting input connector buffering.") changed `ConnectorConfig::max_queued_records` from u64 to `Option<u64>`. This should have been a helpful change because it allows the user to configure only the new `max_queued_bytes` setting, since bytes are more meaningful than records for the purpose of limiting the amount of input buffering, but it caused surprising behavior because a configuration that did not include `max_queued_records` was now interpreted differently from before, which triggered bootstrapping at startup. The best fix for the bootstrapping problem would be for the code that computes differences between pipelines to be updated so that it doesn't treat `max_queued_records` (and some other fields) as significant for the purpose of bootstrapping. That preferred fix is filed as #6427. However, that change would take some consideration, so instead this commit just falls back to the previous definition of `max_queued_records`. This commit may be reverted once #6427 is fixed. Signed-off-by: Ben Pfaff <blp@feldera.com>
3271680 to
776a7d0
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
…tion`.
Commit 753cdd8 ("Add support for bytewise limiting input connector buffering.") changed
ConnectorConfig::max_queued_recordsfrom u64 toOption<u64>. This should have been a helpful change because it allows the user to configure only the newmax_queued_bytessetting, since bytes are more meaningful than records for the purpose of limiting the amount of input buffering, but it caused surprising behavior because a configuration that did not includemax_queued_recordswas now interpreted differently from before, which triggered bootstrapping at startup.The best fix for the bootstrapping problem would be for the code that computes differences between pipelines to be updated so that it doesn't treat
max_queued_records(and some other fields) as significant for the purpose of bootstrapping. That preferred fix is filed as #6427.However, that change would take some consideration, so instead this commit just falls back to the previous definition of
max_queued_records. This commit may be reverted once #6427 is fixed.