Repository navigation
Add Query Library - #18826
Add Query Library#18826nakulnagargade wants to merge 88 commits into
Conversation
- Updated UBIQUITOUS_LANGUAGE.md to clarify definitions related to Query Libraries and Library Queries. - Improved architecture-map.md to reflect changes in how library queries are processed. - Enhanced product-map.md to provide clearer descriptions of Query Libraries and their usage. - Implemented a new QueryLibraryRedirect component for better navigation to the Query Library. - Added LibraryUsagePanel and LibraryVersionBar components to manage and display library query usage and versioning. - Introduced ReleaseConfirmation component updates to include library version notices during app releases. - Added tests for new components and functionalities to ensure reliability and correctness. - Updated styles for Query Library components for improved UI consistency. This commit lays the groundwork for a more robust Query Library feature, enhancing user experience and functionality.
Security Audit ReportNode module vulnerabilities summary (total across frontend, server, marketplace, plugins, root): Critical packages:
|
Security Audit ReportNode module vulnerabilities summary (total across frontend, server, marketplace, plugins, root): Critical packages:
|
CI —
|
| Step | Result |
|---|---|
| Lint & Build | âś… server · âś… plugins · âś… frontend · âŹď¸Ź marketplace |
| Frontend tests | âś… |
| Unit tests | ✅ 1930 passed — step ↗ |
| E2E tests | ✅ 1499 passed — step ↗ |
| Cypress — Platform | ✅ |
| Cypress — Marketplace | âŹď¸Ź skipped |
| Vulnerability gate | ❌ failure |
| Coverage — server | âŹď¸Ź skipped — needs green server suites |
❌ Not mergeable — fix the failures, push, then re-add run-ci.
Full pipeline — required to merge into protected branches. Valid for this commit only.
- Check library references only when a write sets or changes them; carried references (import, git pull, version create) are kept when the library query is missing and fail at run time with a clear message; release and promote still refuse them - Soft licence like Modules: CE and unlicensed workspaces export and import apps that reference the library - Rate-limit library run routes; start the template sandbox only when a query has templates and cache compiled moment/lodash - Add foreign-key indexes for library tables and data_queries indexes; rewrite the library static data source data migration set-based - Map Postgres deadlocks to a retryable 409 - Validate library names and cap draft options size - Document the query-library module invariants - Untrack the stray frontend/agent-skills gitlink and .superpowers/
Covers add to library, running across environments, app promote/release gates, missing library queries, import with and without the licence, folders, Used by, and nested library queries.
# Conflicts: # frontend/ee # frontend/src/AppBuilder/Header/VersionManager/VersionManagerDropdown.jsx # frontend/src/AppBuilder/QueryManager/Components/DataSourcePicker.jsx # frontend/src/AppBuilder/QueryManager/Components/DataSourceSelect.jsx # frontend/src/AppBuilder/_stores/slices/dataQuerySlice.js # server/AGENTS.md # server/ee # server/src/modules/apps/service.ts # server/src/modules/apps/util.service.ts # server/src/modules/data-queries/service.ts # server/src/modules/import-export-resources/service.ts # server/src/modules/versions/AGENTS.md # server/src/modules/versions/controller.ts # server/src/modules/versions/interfaces/IController.ts # server/src/modules/versions/service.ts # server/test/modules/data-queries/unit/query-version-scope.spec.ts
Mocks and expectations lagged the import-export two-pass table export, library DI on DataQueriesService, and resolveDashboardBranchId's signature.
Unassigned QueryLibrary sources failed the frontend CI coverage-manifest check.
Prefer fetched version status over a stale selectedVersion DRAFT snapshot, and tighten openApp's getAppData intercept so /api/apps/addable cannot steal the alias.
Wait for the groups list before selecting a custom group, and allow one retry on the end-user folder permissions spec that was flaking after dual onboarding.
Co-authored-by: Cursor <cursoragent@cursor.com> # Conflicts: # cypress-tests/cypress/support/utils/platform/dataSourceFolders.js
Empty visible/development version id arrays filtered every row out of the dropdown while fetches were still in flight, hiding Create draft after editor boot. Retry development fetch once environments hydrate.
npm ci failed when optional html-minifier/mjml were declared in package.json but absent from the lockfile. Also bump server/ee.
|
Dependency limit exceeded — report not shown. This pull request scan exceeded the 10,000-dependency limit applied to this scan, so the results are incomplete and may be inaccurate. To avoid reporting false positives, Socket has not posted a report. Upgrade your plan to raise the dependency limit and get complete reports, or view the partial scan in the dashboard. Socket is always free for open source. If this is a non-commercial open source project, contact us to request a free Team account. |
Locking them to satisfy npm ci pulled a HIGH-vuln email stack into the runtime audit surface. They have no server imports — remove them instead.
The job still used node:22.15.1-bookworm while engines require 22.23.4, so npm ci failed with EBADENGINE before any migration ran.
| // (parent and snapshot must exist). An unchanged reference may point at a library query that is not in this | ||
| // workspace (imported, pulled from git, or deleted): it is locked only if it exists, and running it reports the | ||
| // missing query. | ||
| export async function lockLibraryPins( |
There was a problem hiding this comment.
A builder can pin a library query whose data source they aren't allowed to use, then run it from their app. Pinning only checks that the version exists. Can we also check data source permission here, like the library page does?
| return LibraryQuery; | ||
| } | ||
| protected defineAbilityFor(can: AbilityBuilder<FeatureAbility>['can'], permissions: UserAllPermissions): void { | ||
| if (!permissions.isEndUser) can(Object.values(FEATURE_KEY), LibraryQuery); |
There was a problem hiding this comment.
Any builder can manage the library here, but "Add to library" and import need module-create permission. Let's use one rule.
| return await dbTransactionWrap(async (manager: EntityManager) => { | ||
| let branchId = appCreateDto.branchId; | ||
| // Workflows always use the workspace default branch, independent of dashboard context. | ||
| const isDefaultBranchOnly = type === APP_TYPES.WORKFLOW; |
There was a problem hiding this comment.
This workflow default-branch change isn't part of Query Library. Can we move it to a separate PR?
| const details = await this.gitSyncConfigsUtilService.getDetails(app.organizationId); | ||
| const gitEnabled = details.isEnabled; | ||
| const defaultBranchId = details.options.defaultBranch?.id ?? null; | ||
| const effectiveBranchId = app.type === APP_TYPES.WORKFLOW ? undefined : branchId; |
There was a problem hiding this comment.
Same as the workflow change in apps/service.ts: please move it to a separate PR.
|
|
||
| export const APPS_PAGE_SIZE = 9; | ||
|
|
||
| export const isModuleLikeAppType = (type?: string): boolean => type === APP_TYPES.MODULE; |
There was a problem hiding this comment.
This only checks for MODULE, so it's the same as the old check. Can we drop it and keep === APP_TYPES.MODULE?
| "@xmldom/xmldom": "^0.8.15", | ||
| "proxy-addr": "^2.0.8" | ||
| }, | ||
| "optionalDependencies": { |
There was a problem hiding this comment.
Removing these dependencies isn't related to Query Library. Can we move it to a separate PR?
| console.log(`${MIGRATION_NAME}: [SUCCESS] Add librarydefault data source finished.`); | ||
| } | ||
|
|
||
| public async down(): Promise<void> { |
There was a problem hiding this comment.
This data source stays in every workspace after a rollback, so the PR isn't a two-way door. Can we confirm older releases handle it, or mark the PR one-way?
| libraryQueryId: string; | ||
| libraryVersionId: string; | ||
| }> { | ||
| throw new NotFoundException('Query Library is unavailable'); |
There was a problem hiding this comment.
CE placeholder methods should return 501 (NotImplementedException), not 404. Same for the two below.
| get().dataQuery.queries.modules[moduleId].some((query) => query.name === newName), | ||
| getCurrentModuleQueries: (moduleId = 'canvas') => get().dataQuery.queries.modules[moduleId], | ||
| setQueries: (queries, moduleId = 'canvas') => { | ||
| export const createDataQuerySlice = (set, get) => { |
There was a problem hiding this comment.
This changes save, rename and delete for every app query, not just library ones. Can we mention that in the PR description?
Summary
Submodule PRs: https://github.com/ToolJet/ee-frontend/pull/831 · https://github.com/ToolJet/ee-server/pull/1040
Add a workspace Query Library for creating, versioning, and reusing queries across apps. Library access follows the workspace license, while existing valid licenses retain access automatically.
Evidence
Merge Danger
Door: Two-way
Blast Radius: Query authoring, execution, and app import/export.