Skip to content
This repository was archived by the owner on May 20, 2026. It is now read-only.

chore(): add telemetry event for auto mode routing fallback#3780

Merged
TylerLeonhardt merged 10 commits into
microsoft:mainfrom
Sid200026:Sid200026/auto-router-telemetry-fallback
Feb 19, 2026
Merged

chore(): add telemetry event for auto mode routing fallback#3780
TylerLeonhardt merged 10 commits into
microsoft:mainfrom
Sid200026:Sid200026/auto-router-telemetry-fallback

Conversation

@Sid200026

Copy link
Copy Markdown
Member

Added telemetry when the auto mode router errors out and fallback model is used.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR adds telemetry tracking when the auto mode router fails to return a valid model and the system falls back to using the reserved model. This helps monitor router reliability and understand when fallback behavior occurs.

Changes:

  • Added a new telemetry event automode.routerDecisionFallback with GDPR annotations to track router failures
  • The event captures available models, preferred models, and the chosen fallback model when routing fails
Comments suppressed due to low confidence (1)

src/platform/endpoint/node/automodeService.ts:273

  • The control flow logic has a critical bug. The code after the telemetry block (lines 262-272) will only execute when routing fails (inside the if (!selectedModel) block). However, when routing succeeds, selectedModel will be set and the function will skip all the remaining endpoint creation logic, causing it to reach line 273 without returning a value. This will result in a compilation error since the function is declared to return Promise<IChatEndpoint> but doesn't return anything in the success path. The code after line 261 (including the _applyVisionFallback call and endpoint creation) should be moved outside of the if (!selectedModel) block so it executes in both the fallback path and the success path.
			if (selectedModel && shouldRoute) {
				// If routing was attempted but failed, emit event that we are falling back to the reserved model
				{
					/* __GDPR__
						"automode.routerDecisionFallback" : {
							"owner": "tyleonha",
							"comment": "Reports a fallback event when the router fails to return a valid model and we have to fall back to the reserved model.",
							"availableModels": { "classification": "SystemMetaData", "purpose": "FeatureInsight", "comment": "Comma-separated list of available models for this request" },
							"preferredModels": { "classification": "SystemMetaData", "purpose": "FeatureInsight", "comment": "Comma-separated list of preferred models for this request, ordered by preference with the reserved model first" }
							"chosenModel": { "classification": "SystemMetaData", "purpose": "FeatureInsight", "comment": "The model selected by the router" },
						}
					*/
					this._telemetryService.sendMSFTTelemetryEvent("automode.routerDecisionFallback", {
						"availableModels": availableModels.join(','),
						"preferredModels": preferredModels.join(','),
						"chosenModel": selectedModel.model,
					})
				}
			}
			selectedModel = this._applyVisionFallback(chatRequest, selectedModel, reserveToken.available_models, knownEndpoints);

			// If the session token changed, invalidate all cached endpoints so they get recreated with the new token
			const existingEndpoints = (entry && entry.lastSessionToken === reserveToken.session_token) ? entry.endpoints : [];
			let autoEndpoint = existingEndpoints.find(e => e.model === selectedModel.model);
			if (!autoEndpoint) {
				autoEndpoint = this._instantiationService.createInstance(AutoChatEndpoint, selectedModel, reserveToken.session_token, reserveToken.discounted_costs?.[selectedModel.model] || 0, this._calculateDiscountRange(reserveToken.discounted_costs));
				existingEndpoints.push(autoEndpoint);
			}
			this._autoModelCache.set(conversationId, { endpoints: existingEndpoints, tokenBank: reserveTokenBank, lastSessionToken: reserveToken.session_token, lastRoutedPrompt: prompt });
			return autoEndpoint;
		}

Comment thread src/platform/endpoint/node/automodeService.ts Outdated
Comment thread src/platform/endpoint/node/automodeService.ts Outdated
Comment on lines +243 to +260
if (selectedModel && shouldRoute) {
// If routing was attempted but failed, emit event that we are falling back to the reserved model
{
/* __GDPR__
"automode.routerDecisionFallback" : {
"owner": "tyleonha",
"comment": "Reports a fallback event when the router fails to return a valid model and we have to fall back to the reserved model.",
"availableModels": { "classification": "SystemMetaData", "purpose": "FeatureInsight", "comment": "Comma-separated list of available models for this request" },
"preferredModels": { "classification": "SystemMetaData", "purpose": "FeatureInsight", "comment": "Comma-separated list of preferred models for this request, ordered by preference with the reserved model first" }
"chosenModel": { "classification": "SystemMetaData", "purpose": "FeatureInsight", "comment": "The model selected by the router" },
}
*/
this._telemetryService.sendMSFTTelemetryEvent("automode.routerDecisionFallback", {
"availableModels": availableModels.join(','),
"preferredModels": preferredModels.join(','),
"chosenModel": selectedModel.model,
})
}

Copilot AI Feb 16, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The new telemetry for router fallback lacks test coverage. Consider adding a test case in automodeService.spec.ts that verifies the telemetry event is emitted when the router call fails (e.g., throws an error or returns an invalid model) and the system falls back to the reserved model. This would ensure the fallback telemetry is properly captured and that the code path works correctly.

Copilot uses AI. Check for mistakes.
Comment thread src/platform/endpoint/node/automodeService.ts Outdated
Sid200026 and others added 7 commits February 17, 2026 02:32
@TylerLeonhardt

Copy link
Copy Markdown
Member

@Sid200026 aren't you more interested in error telemetry in this case?

@vs-code-engineering vs-code-engineering Bot added this to the February 2026 milestone Feb 19, 2026
@TylerLeonhardt
TylerLeonhardt added this pull request to the merge queue Feb 19, 2026
Merged via the queue into microsoft:main with commit 8f0a331 Feb 19, 2026
9 checks passed
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants