Repository navigation
Conversation
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
Critical issues remain: appending expectations can break Python continuation lines, and unvalidated result values can be misparsed or introduce newlines that damage source code.
2 open issues
What changed in this PR
Adds inline-expectation learning for direct .ql tests using MakeTest, reusing the existing learnEdits protocol while keeping direct-query reconciliation separate from .qlref postprocessing.
Changes:
- Extracts shared comment rendering and edit emission.
- Adds a conservative direct-learning plan and explicit expected-tag alias support.
- Adds Python and Java fixtures with matching direct-test expectations.
| File | Description |
|---|---|
shared/​util/​codeql/​util/​test/​InlineExpectationsTest.qll |
Adds direct learning and shared edit-emission helpers. |
python/​ql/​test/​library-tests/​InlineExpectationsTest/​learn-edits/​Test.ql |
Defines a direct test with tag aliases. |
python/​ql/​test/​library-tests/​InlineExpectationsTest/​learn-edits/​test.py |
Supplies matching alias and foreign annotations. |
python/​ql/​test/​library-tests/​InlineExpectationsTest/​learn-edits/​Test.expected |
Records empty expected output for the Python fixture. |
java/​ql/​test/​library-tests/​InlineExpectationsTest/​learn-edits/​Test.ql |
Defines a direct test for a source method. |
java/​ql/​test/​library-tests/​InlineExpectationsTest/​learn-edits/​Test.java |
Supplies a method with a matching annotation. |
java/​ql/​test/​library-tests/​InlineExpectationsTest/​learn-edits/​Test.expected |
Records empty expected output for the Java fixture. |
Review effort: Balanced (auto)
Comment on lines
+627
to
+628
| parseRelativeUrl(Impl::getRelativeUrl(actualResult.getLocation()), relativePath, line, _, | ||
| line, _) and |
| ) and | ||
| parseRelativeUrl(Impl::getRelativeUrl(actualResult.getLocation()), relativePath, line, _, | ||
| line, _) and | ||
| text = actualResult.getExpectationText() |
Expose the learnEdits protocol from direct MakeTest queries without relying on query metadata or @id. Share edit emission with qlref postprocessing while restricting direct reconciliation to safe line-comment syntax and leaving unsupported forms for verification. Add Python and Java fixtures covering custom expected tags, foreign tags, standard tags, and missing query IDs. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: e860263a-9e7b-4451-8b4a-3b5d119a962c
Require generated expectation atoms to round-trip through the inline grammar before emitting an edit. Let language implementations reject unsafe end-of-line appends, and conservatively disable Python appends where explicit continuation lines cannot be distinguished at this seam. Cover unsafe hash, whitespace, newline, and Python continuation cases through the public learnEdits relation. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 1adfa307-c48e-43f5-94e3-8286e4a8c083
d10c
force-pushed
the
d10c/direct-ql-maketest-learning
branch
from
November 4, 2026 14:01
2a07a02 to
ae1a0d7
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does it solve?
Inline-expectation learning needs to work for direct test
.qlqueries that useMakeTest, not only for.qlreftests with a postprocessor.Why is the problem a problem?
Direct tests have no
.qlrefownership metadata or postprocessing stage. Without an explicit direct-query protocol, they cannot emit source edits while preserving their ordinary result relations.How does the solution work?
Direct learning plans emit the same six-column
learnEditsrelation consumed by the test runner. Exact main-schema recognition avoids treating metadata or@idqueries as edit producers. Shared edit emission is reused, while direct and postprocessing plans remain separate because their discovery, identity, output, and reconciliation contracts differ.Why does the solution solve the problem?
The direct query computes edits from its own
MakeTestexpectations and returns them through the runner's established relation shape. Ordinary query results remain available for expected-output comparison.Why is this a good solution?
It adds direct support without pretending that direct queries and tagged
.qlrefcohorts have interchangeable membership semantics. Mixed or multiple producers can therefore be rejected safely until a generalized source-owner contract is designed.Testing
Python and Java library fixtures cover direct edit generation and ordinary result preservation. They were exercised with plain, learn, repeated-learn, and plain verification runs.
Stack: follows #23012 in Stack #23015.