MCP-R ADAR: A Multi-Dimensional Benchmark for

Evaluating Tool Use Capabilities in Large Language

Models

Xuanqi Gao∗ Siyi Xie∗

Xi’an Jiaotong University Xi’an Jiaotong University
arXiv:2505.16700v1 [[Link]] 22 May 2025

Xi’an, China Xi’an, China

gxq2000@[Link] xsy2225025620@[Link]

Juan Zhai Shqing Ma

University of Massachusetts at Amherst University of Massachusetts at Amherst
Amherst, USA Amherst, USA
juanzhai@[Link] shiqingma@[Link]

Chao Shen
Xi’an Jiaotong University
Xi’an, China
chaoshen@[Link]

Abstract

As Large Language Models (LLMs) evolve from passive text generators to ac-
tive reasoning agents capable of tool interaction, the Model Context Protocol
(MCP) has emerged as a standardized framework for dynamic tool discovery
and orchestration. Despite widespread industry adoption, existing evaluation
methodologies fail to adequately assess tool utilization capabilities within this
new paradigm. This paper introduces MCP-R ADAR, the first comprehensive
benchmark specifically designed to evaluate LLM performance in the MCP frame-
work through a novel five-dimensional approach measuring: answer accuracy,
tool selection efficiency, computational resource efficiency, parameter construc-
tion accuracy, and execution speed. Unlike conventional benchmarks that rely
on subjective human evaluations or binary success metrics, MCP-R ADAR em-
ploys objective, quantifiable measurements across multiple task domains including
software engineering, mathematical reasoning, and general problem-solving. Our
evaluations of leading commercial and open-source LLMs reveal distinctive capa-
bility profiles with significant trade-offs between accuracy, efficiency, and speed,
challenging traditional single-metric performance rankings. Besides, we provide
valuable guidance for developers to optimize their tools for maximum model
compatibility and effectiveness. While focused on MCP due to its standardized
approach, our methodology remains applicable across all LLM agent tool integra-
tion frameworks, providing valuable insights for both LLM developers and tool
creators to optimize the entire LLM-tool interaction ecosystem. The implementa-
tion, configurations, and datasets used in our evaluation are publicly available at
[Link]

∗
These authors contributed equally to this work.

Preprint. Under review.

1 Introduction
Large Language Models (LLMs) are rapidly evolving from passive text generators to active reasoning
agents capable of interacting with external tools and APIs. This transformation, driven by function
calling capabilities, has recently culminated in the introduction of the Model Context Protocol (MCP)
- a standardized protocol for LLM-tool interaction that enables dynamic discovery and orchestration
of tools based on task requirements. As MCP adoption accelerates across industry leaders such as
Anthropic [1], OpenAI [2], and Alibaba [3], there is a pressing need for standardized evaluation
frameworks to assess how effectively different models leverage this new paradigm.
The Model Context Protocol represents a significant architectural advancement over previous tool
integration approaches. Unlike traditional function calling, which requires predefined tool configura-
tions and manual API connections, MCP provides a unified framework where AI applications can
dynamically discover, select, and orchestrate tools through a standardized interface. This approach
not only simplifies development workflows but fundamentally transforms how LLMs interact with
external systems - enabling more flexible, context-aware tool utilization that adapts to specific task
requirements rather than following rigid, predetermined pathways.
Existing evaluation methodologies for LLMs primarily focus on knowledge-based reasoning [4, 5],
instruction following [6, 7], and creative generation [8, 9], with limited attention to tool utilization
capabilities. Furthermore, available benchmarks typically rely on subjective human evaluations or
oversimplified binary success metrics that fail to capture the nuanced aspects of effective tool use.
This limitation is particularly problematic for MCP evaluations, where models must not only select
appropriate tools but also construct valid parameters, efficiently orchestrate multi-step workflows,
and transform tool outputs into coherent responses - all while optimizing computational resources.

Radar Bench Dataset Construction Radar Test Implementation

LLMs
General Prompt
(Prompt, Solution, Metrics Analysis)
Assuming all research articles in Nature 2020 relied
Web-browsing on statistical significance (average p-value of 0.04) to
Smithery Glama [Link]
Multi-modality support their conclusions, calculate the number of
MCP Radar

papers incorrectly claiming statistical significance

File-managing (round up).
Answers Answers
Math w/ MCP w/o MCP Solution
Calculating Analysis 1️⃣Let me break down my approach: thought…
Equation Solving RA 2️⃣Identifying the Appropriate Tool: search/fetch tool
CRE RTE
3️⃣Conclusion: calculate:<answer>41</answer>
Code
TFTD TIOR
Code Generation Analysis
Debugging Ground_truth:41/Prediction:41
Code Review LLM’s ability LLMs’ ability Success:True
comparation Tool_usage: tool_calls, arguments…
MCP-Pool in using MCPs
Token_usage: prompt/completion/total_tokens

Figure 1: Overview of MCP-R ADAR.

In this paper, we introduce MCP-R ADAR, the first comprehensive benchmark specifically designed
to evaluate LLM performance within the MCP paradigm, as shown in Figure 1. MCP-R ADAR
employs a novel five-dimensional evaluation framework that quantifies model capabilities across: (1)
answer accuracy, (2) tool selection efficiency, (3) computational resource efficiency, (4) parameter
construction accuracy, and (5) execution speed. This multi-dimensional approach enables researchers
and practitioners to generate intuitive radar visualizations that highlight the relative strengths and
weaknesses of different models across diverse task domains. Our primary objective is not to establish
definitive performance rankings, but rather to provide valuable intuition for both LLM developers
and MCP ecosystem contributors. For LLM developers, MCP-R ADAR offers detailed insights into
model proficiencies and deficiencies across multiple dimensions, guiding targeted improvements
in areas such as tool selection, parameter handling, and resource efficiency. For MCP developers,
our benchmark illuminates how different tool designs and implementation patterns impact model
performance, facilitating the creation of more model-friendly tools and documentation. By bridging
these communities through shared evaluation metrics, we aim to foster collaborative optimization of
the entire LLM-tool interaction experience.
MCP-R ADAR differs from existing LLM benchmarks in several key aspects. First, it focuses
exclusively on tool-augmented capabilities within the standardized MCP framework, allowing for fair
comparison across models regardless of their underlying architectures. Second, it employs strictly
objective metrics derived from quantifiable performance data rather than subjective human judgments,
enhancing reproducibility and reducing evaluation bias. Third, while our initial implementation spans

2
multiple task domains (software engineering, mathematical reasoning, and general problem-solving),
the framework is designed with inherent extensibility, allowing seamless integration of new domains,
tasks, and tools as the ecosystem evolves. Finally, it visualizes performance through intuitive radar
charts that enable rapid identification of model strengths and weaknesses across multiple dimensions.
While we use MCP as our focal point due to its growing prominence and standardized approach,
our evaluation methodology is designed to be universally applicable across all LLM agent tool
integration frameworks. Whether tools are integrated via function calling, plugin architectures,
or other mechanisms, the fundamental capabilities required for effective tool utilization remain
consistent. Our framework measures these core capabilities through objective, quantifiable metrics
that transcend specific implementation details, making it equally valuable for evaluating any tool-
augmented language model regardless of the underlying integration method.
Our research results indicate that tool use performance varies significantly across different models and
task domains, with important trade-offs between accuracy, efficiency, and speed. Through evaluation
of seven mainstream LLMs across three core task domains, we found that model performance
shows clear domain sensitivity, with no “universal champion” existing. Large models generally
perform better in mathematical reasoning domains than in other areas, with an average accuracy
of 0.78, while software engineering and general problem-solving domains only achieved 0.42 and
0.28 respectively. Claude 3.7 and Gemini 2.5 Pro lead in accuracy dimensions, particularly Gemini
2.5 Pro which reached 0.91 accuracy in the mathematics domain, the highest among all models;
however, in efficiency metrics, Gemini and Claude perform relatively weakly, demonstrating different
performance trade-off strategies. GPT-4o displays a more balanced pentagon profile, and while not
optimal in any dimension, it has no obvious shortcomings; open-source models like DeepSeek-V3,
though lagging in accuracy, are the most resource-efficient models, offering significant advantages
in resource-constrained scenarios. Particularly noteworthy is that all models show an expanding
gap between intermediate process performance (TFTD and FEP values) and accuracy in the general
problem-solving domain, indicating that models can partially correctly execute tool calling steps but
struggle to integrate these steps into correct final results, suggesting that current LLMs still have
significant room for improvement in tool chain integration capabilities for open-ended tasks.
Furthermore, our analysis reveals critical insights about the relationship between model capabilities
and tool design patterns. We identify specific tool implementation strategies that consistently enhance
model performance across multiple dimensions, as well as common pitfalls that create unnecessary
barriers to effective tool utilization. These findings provide valuable guidance for developers seeking
to optimize their tools for maximal model compatibility and effectiveness.
Our main contributions include:

• The first systematic evaluation framework for tool-augmented language models that employs
purely objective, quantifiable metrics across multiple performance dimensions.
• MCP-R ADAR, a comprehensive benchmark dataset spanning three critical domains: soft-
ware engineering, mathematical reasoning, and general problem-solving, with built-in
extensibility for additional domains.
• Empirical evaluation of leading commercial and open-source LLMs, revealing distinctive
capability profiles that challenge conventional single-metric performance rankings.
• Detailed analysis of performance patterns and their implications for both model training and
tool design, providing actionable insights for ecosystem optimization.
• Our implementation, configurations and collected datasets are available at [10].

2 Related work
2.1 Agent, tool use and MCP

The Model Context Protocol (MCP) for Large Language Models (LLMs) is a unified interaction
standard proposed by Anthropic to address systematic challenges in tool invocation. Early LLMs such
as the GPT family relied only on static training data and were unable to access real-time information
or interact with external systems, resulting in limited [Link] et al. [11] demonstrated the
role of structured reasoning in improving the performance of LLMs, which provided the basis for

3
subsequent tool invocation. With the increase of complex scenarios such as multi-round dialog
systems, developers try to connect to external via API [12] [13] [14] [15] to external data sources.
Use the tool-enhanced LLM [16] [17] [18] to try to solve the web browsing [19] [20] [21] or code
interpretation [22] [23] and other aspects of relevance, but schick et al. [24] points out that this “peer-
to-peer” integration leads to NxM issues, limiting system expansion and increasing maintenance
costs. While platforms such as Hugging Face promote model sharing, and frameworks such as
LangChain attempt to enhance model capabilities through Tool Calling, these solutions do not address
the underlying problem. However, these solutions still fail to address the fundamental problem. Hsieh
et al. [25]noting that these approaches still lack a common context delivery mechanism. It is in this
context that the Model Context Protocol (MCP) was formally introduced and open sourced.

2.2 Tool/MCP use evaluation

Evaluating the tool-use capabilities of large language models has become an important direction of
research in recent years. While traditional LLM assessment frameworks focus on language compre-
hension capabilities, with the development of tool-enhanced AI [26] [24], assessment benchmarks
specialized in tool-usage capabilities have become especially necessary [27] [28] [16]. The HELM
framework proposed by Liang et al. [29] attempts to establish multidimensional evaluation criteria,
but still does not focus on the efficiency and effectiveness of tool interaction protocols. Existing tool
usage evaluation either struggles to handle more complex task scenarios such as long context memory
calls, multiple rounds or multi-tool call tests [30] [16] [27][22][15][14], or a single path of standard-
ized answers to task scenarios that do not match real-world user needs [31]. At the same time, all of
these evaluation systems lack a systematic assessment of the model’s ability to use a specific protocol
(e.g., MCP). At the practical application level, there is a lack of unified evaluation criteria to measure
the model’s fitness with MCPs, the performance impact of MCPs on large models, and the efficiency
of multi-MCP collaboration. In this paper, we introduce MCP-R ADAR, a large-scale instruction
benchmark, to explore the performance of LLMs in a variety of real-world MCP usage scenarios, and
introduce mathematical, code, and generalized ability tests for MCP tools, incorporating metrics such
as accuracy, average round accuracy, and task processing rate for a more comprehensive evaluation.

3 MCP-R ADAR

3.1 MCP-R ADAR Overview

Current methods for evaluating large language models’ tool utilization capabilities often rely on
single metrics or subjective judgment, making comprehensive performance assessment difficult.
We introduce MCP-R ADAR, a framework based on three core principles: objective quantifiabil-
ity, multidimensional complementarity, and domain universality. MCP-R ADAR assesses model
performance across five key dimensions:
Result Accuracy (RA) Tool Call Process Accuracy (TCPA) First Error Position (FEP) Computational
Resource Efficiency (CRE) Response Time Efficiency (RTE)
We’ve developed a benchmark test set covering software engineering, mathematical reasoning, and
general problem-solving domains. Our framework visualizes model performance through radar charts,
providing intuitive comparison of different models’ capabilities. Additionally, we design experiments
to analyze model sensitivity to tool documentation quality, offering empirical guidance for MCP
ecosystem optimization.

3.2 Evaluation Dimensions

MCP-R ADAR employs five complementary dimensions((RA), (DTSR), (FEP), (CRE) and (RTE))
to comprehensively evaluate large language models’ tool utilization capabilities in MCP environments,
with each dimension capturing distinct critical aspects of performance.
The detailed information and calculation methods for these evaluation metrics can be found in Ap-
pendix D.

4
3.3 Benchmark Dataset

To comprehensively evaluate LLMs’ tool utilization capabilities in MCP environments, we have

constructed a benchmark test set encompassing diverse tasks and tools. This test set is meticulously
designed to ensure broad domain coverage, task complexity gradients, and tool description variants,
enabling systematic analysis of model performance.

3.3.1 Task Domains and Distribution

Our benchmark test set contains 300 tasks across three core domains, with 100 tasks per domain:

• Software Engineering: This domain includes tasks related to code generation, debugging,
and code review, requiring models to interact with programming tools and APIs.

• Mathematical Reasoning: This domain encompasses tasks that involve mathematical

problem-solving, requiring models to utilize mathematical tools and libraries.

• General Problem Solving: This domain encompasses practical, everyday tasks including
web search, information gathering, research queries, and other common real-world problems
that require tool-assisted information retrieval and processing.

These tasks are selected from high-quality open-source datasets (such as GAIA [32], GSM8k [8],
MATH [9], HumanEval [33], MBPP [34], etc.) and carefully reconstructed to ensure their suitability
for MCP tool usage evaluation. Each task undergoes standardized processing, including clear
inputs, expected outputs, and scoring criteria, thereby supporting automated assessment and result
verification. Each task undergoes multiple rounds of validation to ensure quality:
1. Initial design stage by domain experts creating tasks and annotating standard answers;
2. Independent verification stage by two different reviewers completing tasks, confirming solution
paths and standard answer clarity;
3. Consistency checks ensuring multiple reviewers obtain identical results using the same tool set,
eliminating ambiguous tasks;
4. Difficulty calibration processes ensuring consistency of difficulty levels within domains
This rigorous process ensures high quality and reliability of the benchmark test, providing a solid
foundation for model evaluation. Each task in the test set is equipped with clear inputs, expected
outputs, and scoring criteria, facilitating automated assessment.

3.3.2 Task Complexity Grading

To better evaluate large model capabilities, we classify tasks by complexity from Level 1 (simplest)
to Level 3 (most complex), based on the following criteria:
Level 1 Tasks require single tool calls, simple parameters, no complex reasoning
Level 2 Tasks require 2-4 tool calls, parameters needing moderate processing, multi-step reasoning.
Level 3 Tasks require more than 4 tool calls, parameters needing complex processing, deep reasoning
and planning.
The distribution of tasks by complexity in each domain is shown in Figure 2.

3.3.3 MCP Tool Analysis

We have collected 42 mainstream MCP, covering software development tools (GitHub, VSCode, file
systems, etc.), mathematical reasoning tools (calculators, expression solvers, etc.), and general tools
(search engines, knowledge bases, etc.). These tools have all been applied in actual MCP ecosystems,
representing current mainstream tool usage scenarios.

5
 'LVWULEXWLRQRI7DVN&RPSOH[LW\$FURVV'RPDLQV

/HYHO6LPSOH
/HYHO0HGLXP
/HYHO&RPSOH[







1XPEHURI7DVNV 

 



 
 


6RIWZDUH 0DWKHPDWLFDO *HQHUDO
(QJLQHHULQJ 5HDVRQLQJ 3UREOHP6ROYLQJ
'RPDLQ

Figure 2: Tasks distribution.

Domain #Tools
Tool Library 2
Classification Repository 2
Calculation 5
ProjectManagement 4
WorkflowExecution 6
WebSearch 4
Terminal 8
File 7
Directory 4

Figure 3: The distribution of tools. Figure 4: Tool statistics.

4 Experiment
4.1 Setup

We designed a series of comparative experiments to test the performance of different models across
various task scenarios and tool configurations. This section details the experimental setup, including
model selection, environment configuration, evaluation protocols, and data collection methods.
We selected 7 representative large language models for evaluation, covering current mainstream
commercial closed-source models (Claude 3.7 Sonnet [35], GPT-4o [36], GPT-4o-mini [37], and
Gemini 2.5 Pro [38]) and open-source models (Llama 3.3 70B [39], DeepSeek V3 [40], and Qwen 3
245B [41]). The model selection covers different architectures, scales, and specialization degrees,
enabling us to comprehensively compare the performance differences of various models in the MCP
environment. Each model uses the latest version officially provided, ensuring the experimental results
represent the current technological level.
Additionally, we built a standardized MCP test environment to ensure all models are evaluated
under the same conditions. MCP servers provide services through standard protocols, with each

6
 Table 1: Comparison of model performance metrics across three domains of MCP-R ADAR

Task Model RA↑ TFTD↑ FEP↑ CRE↓ RTE↓

Claude 3.7 0.89 0.95 0.93 5925 6441
Gemini 2.5 Pro 0.91 0.97 0.92 6187 4171
GPT-4o 0.85 0.92 0.90 5865 4725
Math GPT-4o-mini 0.72 0.88 0.86 5024 2845
Qwen3 0.85 0.95 0.95 5365 11053
Deepseek-V3 0.59 0.91 0.88 1813 2975
Llama3 0.64 0.80 0.77 4838 3733
Claude 3.7 Sonnet 0.58 0.57 0.78 2529 4212
Gemini 2.5 Pro 0.54 0.55 0.69 2894 2424
GPT-4o 0.46 0.48 0.58 3481 3533
Coding GPT-4o-mini 0.32 0.41 0.62 1180 2143
Qwen3 0.52 0.60 0.58 1182 4506
Deepseek-V3 0.33 0.36 0.44 1205 1451
Llama3 0.21 0.22 0.24 1506 1471
Claude 3.7 Sonnet 0.33 0.72 0.72 5214 3539
Gemini 2.5 Pro 0.46 0.68 0.64 6255 4550
GPT-4o 0.27 0.31 0.44 5603 4450
General GPT-4o-mini 0.13 0.29 0.33 4883 1077
Qwen3 0.28 0.42 0.40 4854 12135
Deepseek-V3 0.25 0.56 0.50 5183 1511
Llama3 0.22 0.35 0.28 2213 2014

server offering several related tools. All servers are deployed in the same hardware environment to
ensure consistency in response time. We use Openrouter as the model interface provider to ensure
comparability of response times between models, and maintain unchanged interface configurations
during the experiment to eliminate potential impacts from API changes.

4.2 Main Results

Table 1 shows the performance of 7 mainstream large language models in the three core task domains
of MCP-R ADAR, including Result Accuracy (RA), Dynamic Tool Selection Rate (DTSR), First Error
Position (FEP), Computational Resource Efficiency (CRE), and Response Time Efficiency (RTE).
Lower values for CRE and RTE indicate better efficiency.

Domain Performance Variations. Models demonstrate significant performance differences across

domains. In the mathematical reasoning domain, all models achieved the highest RA, averaging 0.78,
substantially higher than software engineering (0.42) and general problem-solving domains (0.28).
This indicates current LLMs have more reliable tool usage capabilities in structured, well-defined
mathematical tasks. Gemini 2.5 Pro achieved the highest mathematics RA at 0.91, with Claude 3.7
following at 0.89. Similarly, TCPA scores averaged 0.91 in mathematics versus 0.46 and 0.48 in other
domains. Qwen3 demonstrated exceptional stability in mathematical tasks (TCPA=0.95), despite
slightly lower final accuracy (RA=0.85).

Efficiency Tradeoffs. Computational efficiency varies dramatically between models, revealing

potential accuracy-efficiency tradeoffs. DeepSeek-V3 demonstrates superior efficiency (CRE=1813
in mathematics), requiring significantly fewer tokens than Gemini 2.5 Pro (CRE=6187), but with
lower accuracy (RA=0.59). For response time, GPT-4o-mini excels across all domains (RTE=2143 in
software engineering; RTE=1077 in general problems), making it suitable for real-time applications.
Conversely, Qwen3 exhibits significantly slower response times (RTE=12135 in general problems)
despite competitive accuracy.

Model Performance Profiles. Figure 5 visualizes each model’s multidimensional performance

characteristics. In mathematical reasoning, Gemini 2.5 Pro and Claude 3.7 show strong performance
in RA and TCPA dimensions, though Claude 3.7 demonstrates weaker computational efficiency and
response speed. GPT-4o displays a balanced pentagon profile with no notable weaknesses. In contrast,

7
 Coding General Math

Figure 5: Model Performance Comparison Across Tasks. Longer edges indicate superior performance
in each metric.

Llama3 shows higher computational efficiency but lower accuracy. In the coding domain, all profiles
generally contract, with average RA dropping to 0.42, indicating increased difficulty in tool usage.
The general problem-solving domain reveals the most significant model differences, with Gemini 2.5
Pro’s RA (0.46) outperforming the second-place Claude 3.7 (0.33) by approximately 40%.

Performance Tier Analysis. Based on the radar charts, models can be classified into three perfor-
mance tiers: (1) Leading tier: Gemini 2.5 Pro and Claude 3.7, with highest accuracy but different
efficiency focuses; (2) Middle tier: GPT-4o, Qwen3, and GPT-4o-mini, each with specific strengths
and weaknesses; (3) Following tier: DeepSeek-V3 and Llama3, lagging in accuracy but competitive
in certain efficiency metrics. This classification reflects the current development stages of LLMs in
MCP environments.
Llama 3.3 70B Qwen 3 245B DeepSeek V3

Open Source

Close Source

GPT4o-mini GPT4o Claude 3.7 Sonnet Gemini 2.5 Pro

Figure 6: Comprehensive Model Metrics. Longer edges indicate superior performance in each metric.

8
Open-Source vs. Closed-Source Comparison. Figure 6 contrasts model performance by type.
Open-source models (top) show distinct optimization patterns: Llama3 prioritizes response time but
has poor accuracy, Qwen3 excels in mathematical accuracy but struggles with speed, and DeepSeek-
V3 demonstrates superior computational efficiency. Closed-source models (bottom) generally exhibit
larger profile areas and greater cross-domain stability. GPT-4o-mini and GPT-4o show balanced
profiles, while Claude 3.7 and Gemini 2.5 Pro demonstrate stronger accuracy at the expense of
efficiency.

Cross-Domain Generalization. The most significant difference between model groups lies in
profile stability across domains. Closed-source models maintain more consistent performance when
transitioning between domains, while open-source models show more pronounced performance drops.
For example, Llama3’s RA drops by 67% from mathematics (0.64) to programming (0.21), while
GPT-4o decreases by only 46% (from 0.85 to 0.46). This suggests closed-source models have more
robust generalization capabilities across diverse tool-use scenarios.

5 Discussion
Based on the results of MCP-R ADAR’s comprehensive assessment, we have come up with the
following key insights and takeaways:
There is no “all-rounder” model in the MCP environment now. Although Claude 3.7 and Gemini 2.5
Pro lead in overall performance, they show different patterns of dominance in different domains. For
example, Claude 3.7 had extremely high First Error Position (FEP) and Result Accuracy (RA) in the
math task, but relatively average performance in the general problem domain. This suggests that the
model exhibits significant “domain specialization” in the MCP task. This domain specificity may
stem from two key factors: (1) differences in the proportion of different domains in the pre-training
data distribution; and (2) the model architecture’s intrinsic preference for specific types of reasoning.
Notably, all models generally outperformed the software engineering and generalized problem solving
domains in the mathematical domain, which may reflect the fact that the deterministic nature of
mathematical tasks is more in line with the capability boundaries of current LLMs.
Fast first, then accurate. The experimental data show that instead of strictly pursuing full accuracy
(high DTSR) of process calls, it is better to allow the model to try and correct errors more quickly,
thus significantly reducing response time (improving RTE), especially in scenarios with frequent user
interactions. We observe that GPT-4o-mini, although lower in accuracy than its full version, has a
significant response time advantage and may provide a better user experience in real-time interaction
scenarios. This provides important insights for MCP tool developers: when designing tool interfaces
and error handling mechanisms, they should prioritize fast attempts and fast recovery paths rather
than pursuing error-free execution.
Overemphasizing the accuracy of each step may affect the overall progress of the task. According
to the DTSR and FEP metrics, certain models (e.g., GPT-4o) showed high in-process call accuracy,
but did not show a significant advantage in first error occurrence position (FEP). This suggests that
in highly accurate short interactions, models may instead be overly cautious or conservative, and
more prone to early errors or the need for frequent error correction for complex tasks. For LLM
developers, this means that the training goal should be to strike an appropriate balance between “local
accuracy” and “global efficiency”. For example, by increasing the model’s tolerance for uncertainty,
allowing possible solution paths to be tried with a low degree of certainty rather than being abandoned
prematurely.
Error recovery mechanism is important. Interestingly, high FEP values (later errors) do not always
lead to significant accuracy improvement. From the experimental data, some models (e.g., DeepSeek-
V3 in the software engineering domain) have high FEP values, implying that the model’s first error
occurs at a later stage of the task process, yet accuracy is not significantly improved. This finding
suggests that in MCP environments, late-stage error repair or fault-tolerance mechanisms may have
a greater impact on task success than simply delaying the timing of errors. This raises important
implications for LLM developers: the focus should be on developing error recovery mechanisms,
especially for complex error states that may occur later in the task, rather than focusing solely on
avoiding errors.
We offer the following practical advice for different stakeholders:

9
For MCP tool developers:

• Design a robust error handling mechanism. Our experimental results show that error handling
ability is a key factor affecting task success, especially in complex tasks. Error reports
should distinguish between parameter errors, execution errors, and result errors, enabling
the model to adopt appropriate strategies for different types of errors; and provide undo or
rollback operations to facilitate model recovery from errors.
• Decomposition of complex parameter structures into simple components. The experimental
results show that the accuracy of parameter construction decreases significantly for all mod-
els as the parameter complexity increases. To reduce the difficulty of model understanding,
it is suggested that avoid deep nested objects, decompose complex parameters into multiple
flat parameters, limit the number of parameters, and provide reasonable default values for
non-essential parameters to reduce the burden of model configuration.

For LLM developers:

• Strengthen the model’s error recovery ability. Current LLMs have significant deficiencies
in error recovery, especially when dealing with late task errors. Developers should design
specialized pre-training tasks to improve the model’s ability to parse and understand error
messages, and to develop error recovery strategies rather than simple retries.
• Optimize inference link consistency. Developers should improve the model’s ability to
maintain consistency over long sequences and ensure that the training data contains sufficient
samples of multiple rounds of tool interactions.

For MCP application developers:

• Adopting a “fast first, then accurate” strategy in real-time interaction scenarios. Quickly
return preliminary results before progressively refining and correcting them, and allow the
model to continue interacting with the user while waiting for time-consuming tools to be
executed.
• Design appropriate intermediate feedback mechanisms for complex tasks. For example,
visualize the current progress and expected time remaining for multi-step tasks and allow
users to confirm or redirect at key decision points.
• Introduction of tool-use memory and adaptive learning mechanisms. For example, record
the tool usage preferences and success patterns of specific users, recommend effective tool
combinations for new tasks based on historical successes, or save the last successful tool
invocation sequence for similar tasks.

With these targeted recommendations, all participants in the MCP ecosystem can work together to
promote the practical application value of tool-enhanced LLMs and drive the technology toward
greater reliability, efficiency and user-friendliness.

6 Conclusion
This study presents MCP-R ADAR, the first multidimensional evaluation framework designed specifi-
cally for assessing the tool usage capabilities of large language models in Model Context Protocol
environments. With five complementary dimensions, we go beyond the traditional single-metric
assessment to provide comprehensive insights into the tool-usage capabilities of models. Based on
these findings, we provide practical guidance for all parties in the MCP ecosystem. MCP-R ADAR
provides a systematic methodology for assessing and improving the ability of LLMs to utilize tools,
revealing the current state of the art and opportunities for improvement. With the development of
this critical capability, we believe that future LLMs will be able to utilize tools more efficiently and
accurately, providing reliable value for a wide range of real-world application scenarios.

References
[1] Introducing the Model Context Protocol. URL [Link]
model-context-protocol.

10
 [2] Model context protocol (MCP) - OpenAI Agents SDK. URL [Link]
openai-agents-python/mcp/.
[3] QwenLM/Qwen-Agent, . URL [Link]
[4] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and
Jacob Steinhardt. Measuring Massive Multitask Language Understanding, . URL http:
//[Link]/abs/2009.03300.
[5] Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang, Shuai Lu, Yanlin Wang, Amin
Saied, Weizhu Chen, and Nan Duan. AGIEval: A Human-Centric Benchmark for Evaluating
Foundation Models. URL [Link]
[6] Xingyao Wang, Zihan Wang, Jiateng Liu, Yangyi Chen, Lifan Yuan, Hao Peng, and Heng Ji.
MINT: Evaluating LLMs in Multi-turn Interaction with Tools and Language Feedback. URL
[Link]
[7] Mohit Shridhar, Xingdi Yuan, Marc-Alexandre Côté, Yonatan Bisk, Adam Trischler, and
Matthew Hausknecht. ALFWorld: Aligning Text and Embodied Environments for Interactive
Learning. URL [Link]
[8] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser,
Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John
Schulman. Training Verifiers to Solve Math Word Problems. URL [Link]
2110.14168.
[9] Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn
Song, and Jacob Steinhardt. Measuring Mathematical Problem Solving With the MATH Dataset,
. URL [Link]
[10] Anonymized Repository - Anonymous GitHub. URL [Link]
r/MCPRadar-B143/[Link].
[11] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le,
Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models.
Advances in neural information processing systems, 35:24824–24837, 2022.
[12] Huanxi Liu, Jiaqi Liao, Dawei Feng, Kele Xu, and Huaimin Wang. Autofeedback: An llm-based
framework for efficient and accurate api request generation. arXiv preprint arXiv:2410.06943,
2024.
[13] Yifan Song, Weimin Xiong, Dawei Zhu, Wenhao Wu, Han Qian, Mingbo Song, Hailiang
Huang, Cheng Li, Ke Wang, Rong Yao, et al. Restgpt: Connecting large language models with
real-world restful apis. arXiv preprint arXiv:2306.06624, 2023.
[14] Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong,
Xiangru Tang, Bill Qian, et al. Toolllm: Facilitating large language models to master 16000+
real-world apis. arXiv preprint arXiv:2307.16789, 2023.
[15] Qiaoyu Tang, Ziliang Deng, Hongyu Lin, Xianpei Han, Qiao Liang, Boxi Cao, and Le Sun.
Toolalpaca: Generalized tool learning for language models with 3000 simulated cases. arXiv
preprint arXiv:2306.05301, 2023.
[16] Shishir G Patil, Tianjun Zhang, Xin Wang, and Joseph E Gonzalez. Gorilla: Large language
model connected with massive apis. Advances in Neural Information Processing Systems, 37:
126544–126565, 2024.
[17] Aaron Parisi, Yao Zhao, and Noah Fiedel. Talm: Tool augmented language models. arXiv
preprint arXiv:2205.12255, 2022.
[18] Pan Lu, Baolin Peng, Hao Cheng, Michel Galley, Kai-Wei Chang, Ying Nian Wu, Song-Chun
Zhu, and Jianfeng Gao. Chameleon: Plug-and-play compositional reasoning with large language
models. Advances in Neural Information Processing Systems, 36:43447–43478, 2023.

11
[19] Timo Schick and Hinrich Schütze. Exploiting cloze questions for few shot text classification
and natural language inference. arXiv preprint arXiv:2001.07676, 2020.
[20] Michal Spiegel and Aleš Horák. Webmap: Improving llm web agents with semantic search for
relevant web pages.
[21] Gobinda Chowdhury and Sudatta Chowdhury. Ai-and llm-driven search tools: A paradigm
shift in information access for education and research. Journal of Information Science, page
01655515241284046, 2024.
[22] Yuchen Zhuang, Yue Yu, Kuan Wang, Haotian Sun, and Chao Zhang. Toolqa: A dataset for llm
question answering with external tools. Advances in Neural Information Processing Systems,
36:50117–50143, 2023.
[23] Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang
Ding, Kaiwen Men, Kejuan Yang, et al. Agentbench: Evaluating llms as agents. arXiv preprint
arXiv:2308.03688, 2023.
[24] Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Eric Hambro,
Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models
can teach themselves to use tools. Advances in Neural Information Processing Systems, 36:
68539–68551, 2023.
[25] Cheng-Yu Hsieh, Si-An Chen, Chun-Liang Li, Yasuhisa Fujii, Alexander Ratner, Chen-Yu Lee,
Ranjay Krishna, and Tomas Pfister. Tool documentation enables zero-shot tool-usage with large
language models. arXiv preprint arXiv:2308.00675, 2023.
[26] Zekun Wang, Ge Zhang, Kexin Yang, Ning Shi, Wangchunshu Zhou, Shaochun Hao,
Guangzheng Xiong, Yizhi Li, Mong Yuan Sim, Xiuying Chen, et al. Interactive natural
language processing. arXiv preprint arXiv:2305.13246, 2023.
[27] Qiantong Xu, Fenglu Hong, Bo Li, Changran Hu, Zhengyu Chen, and Jian Zhang. On the tool
manipulation capability of open-source large language models. arXiv preprint arXiv:2305.16504,
2023.
[28] Yaobo Liang, Chenfei Wu, Ting Song, Wenshan Wu, Yan Xia, Yu Liu, Yang Ou, Shuai Lu, Lei
Ji, Shaoguang Mao, et al. Taskmatrix. ai: Completing tasks by connecting foundation models
with millions of apis. Intelligent Computing, 3:0063, 2024.
[29] Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga,
Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, et al. Holistic evaluation of
language models. arXiv preprint arXiv:2211.09110, 2022.
[30] Minghao Li, Yingxiu Zhao, Bowen Yu, Feifan Song, Hangyu Li, Haiyang Yu, Zhoujun Li,
Fei Huang, and Yongbin Li. Api-bank: A comprehensive benchmark for tool-augmented llms.
arXiv preprint arXiv:2304.08244, 2023.
[31] Pei Wang, Yanan Wu, Zekun Wang, Jiaheng Liu, Xiaoshuai Song, Zhongyuan Peng, Ken Deng,
Chenchen Zhang, Jiakai Wang, Junran Peng, et al. Mtu-bench: A multi-granularity tool-use
benchmark for large language models. arXiv preprint arXiv:2410.11710, 2024.
[32] Grégoire Mialon, Clémentine Fourrier, Craig Swift, Thomas Wolf, Yann LeCun, and Thomas
Scialom. GAIA: A benchmark for General AI Assistants. URL [Link]
2311.12983.
[33] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto,
Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul
Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke
Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad
Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias
Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex
Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain,
William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra,

12
 Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer,
Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech
Zaremba. Evaluating Large Language Models Trained on Code. URL [Link]
abs/2107.03374.

[34] Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David
Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton. Program Synthesis
with Large Language Models. URL [Link]

[35] Claude 3.7 Sonnet and Claude Code. URL [Link]

claude-3-7-sonnet.

[36] Hello GPT-4o. URL [Link]

[37] GPT-4o mini: Advancing cost-efficient intelligence. URL [Link]

gpt-4o-mini-advancing-cost-efficient-intelligence/.

[38] Gemini 2.5: Our most intelligent AI model. URL [Link]

google-deepmind/gemini-model-thinking-updates-march-2025/.

[39] Llama 3.3 | Model Cards and Prompt formats. URL [Link]
model-cards-and-prompt-formats/llama3_3/.

[40] Chars Yang. DeepSeek v3 - Advanced AI & LLM Model Online. URL [Link]
org/.

[41] Qwen3: Think Deeper, Act Faster | Qwen, . URL [Link]

qwen3/.

A Conversation Logs and Response Process

Example-Code
System Prompt: You are a code assistant that MUST use available tools to solve code problems.
Never try to solve problems directly when tools can help.
Process:
Analyze the problem Choose appropriate tool(s) Call tool(s) with correct parameters Interpret
results and use additional tools if needed Provide final answer
Important:
Always use available tools Show your tool-calling process Format answer as: <answer>[YOUR
FINAL ANSWER]</answer> Use appropriate number formats or LaTeX for mathematical
answers
User: Q1:"Prompt": "def get_positive(l: list): """Return only positive numbers in the list. »>
get_positive([-1, 2, -4, 5, 6]) [2, 5, 6] »> get_positive([5, 3, -5, 2, -3, 3, 9, 0, 123, 1, -10])
[5, 3, 2, 3, 9, 123, 1] """Please use MCP to break down the task to help solve this problem.
Write the solution and save it in a jsonl file named [Link] using the format {"unique_id":
"000000000000031", "Answer": "your_code_here"}"
Agent: <tool>break down the task</tool>
Step 1: Code Implementation
Step 2: Format the Answer for JSONL
Step 3: Save the Answer in a JSONL File
Step 4: Creating the Directory and Writing the File
<tool>Create directory structure</tool> <tool>Write the formatted answer to JSONL file</tool>
finish!

13
Example-General
System Prompt: You are an assistant that MUST use available tools to solve problems. Never
try to solve problems directly when tools can help.
Process:
Analyze the problem Choose appropriate tool(s) Call tool(s) with correct parameters Interpret
results and use additional tools if needed Provide final answer
Important:
Always use available tools Show your tool-calling process Format answer as: <answer>[YOUR
FINAL ANSWER]</answer> Use appropriate number formats or LaTeX for mathematical
answers
User: Q1: "If we assume all articles published by Nature in 2020 (articles, only, not book
reviews/columns, etc) relied on statistical significance to justify their findings and they on
average came to a p-value of 0.04, how many papers would be incorrect as to their claims of
statistical significance? Round the value up to the next integer."
Agent: Step 1: Search for the number of Nature articles published in 2020
<tool>search for information</tool> Step 2: calculate the expected number of incorrect claims
<answer>41</answer> finish!

Example-Math
System Prompt: You are an math assistant that MUST use available tools to solve math
problems. Never try to solve problems directly when tools can help.
Process:
Analyze the problem Choose appropriate tool(s) Call tool(s) with correct parameters Interpret
results and use additional tools if needed Provide final answer
Important:
Always use available tools Show your tool-calling process Format answer as: <answer>[YOUR
FINAL ANSWER]</answer> Use appropriate number formats or LaTeX for mathematical
answers
−29
  12  −29 
User: Q1: "Find 12

7 · 3 − 7 · 3 ."
Agent: Step 1: Calculate the floor of -29/3 <tool>floor</tool> Step 2: Calculate the ceiling of
the expression <tool>ceiling</tool> <answer>2</answer> finish!

14
B Replicated Experiment Results

Result table 1 presents the average metrics across three experimental runs.

Table 2: First run of model performance metrics (Experiment 1)

Task Model RA↑ TFTD↑ FEP↑ CRE↓ RTE↓

Claude 3.7 0.87 0.94 0.92 5890 6380
Gemini 2.5 Pro 0.89 0.96 0.90 6150 4120
GPT-4o 0.84 0.93 0.91 5830 4780
Math GPT-4o-mini 0.71 0.87 0.84 5050 2820
Qwen3 0.83 0.94 0.94 5320 11120
Deepseek-V3 0.57 0.90 0.87 1845 2950
Llama3 0.63 0.79 0.76 4870 3690
Claude 3.7 0.56 0.56 0.76 2540 4170
Gemini 2.5 Pro 0.52 0.54 0.67 2920 2450
GPT-4o 0.45 0.47 0.57 3520 3560
Coding GPT-4o-mini 0.30 0.40 0.60 1210 2170
Qwen3 0.51 0.59 0.57 1200 4540
Deepseek-V3 0.32 0.35 0.43 1230 1470
Llama3 0.20 0.21 0.23 1530 1490
Claude 3.7 0.31 0.70 0.70 5240 3580
Gemini 2.5 Pro 0.44 0.67 0.62 6290 4590
GPT-4o 0.26 0.30 0.43 5640 4490
General GPT-4o-mini 0.12 0.28 0.32 4910 1090
Qwen3 0.27 0.41 0.39 4890 12180
Deepseek-V3 0.24 0.55 0.49 5220 1530
Llama3 0.21 0.34 0.27 2240 2030

Table 3: Second run of model performance metrics (Experiment 2)

Task Model RA↑ TFTD↑ FEP↑ CRE↓ RTE↓

Claude 3.7 0.90 0.96 0.94 5950 6490
Gemini 2.5 Pro 0.92 0.98 0.93 6200 4220
GPT-4o 0.86 0.91 0.89 5880 4680
Math GPT-4o-mini 0.73 0.89 0.87 5010 2860
Qwen3 0.86 0.96 0.96 5390 10990
Deepseek-V3 0.60 0.92 0.89 1790 2990
Llama3 0.65 0.81 0.78 4810 3760
Claude 3.7 0.59 0.58 0.79 2520 4240
Gemini 2.5 Pro 0.55 0.56 0.70 2880 2400
GPT-4o 0.47 0.49 0.59 3450 3510
Coding GPT-4o-mini 0.33 0.42 0.63 1160 2120
Qwen3 0.53 0.61 0.59 1170 4480
Deepseek-V3 0.34 0.37 0.45 1190 1440
Llama3 0.22 0.23 0.25 1490 1460
Claude 3.7 0.34 0.73 0.73 5190 3510
Gemini 2.5 Pro 0.47 0.69 0.65 6230 4520
GPT-4o 0.28 0.32 0.45 5580 4420
General GPT-4o-mini 0.14 0.30 0.34 4860 1070
Qwen3 0.29 0.43 0.41 4830 12100
Deepseek-V3 0.26 0.57 0.51 5160 1500
Llama3 0.23 0.36 0.29 2200 2000

15
 Table 4: Third run of model performance metrics (Experiment 3)

Task Model RA↑ TFTD↑ FEP↑ CRE↓ RTE↓

Claude 3.7 0.90 0.95 0.93 5935 6453
Gemini 2.5 Pro 0.92 0.97 0.93 6211 4173
GPT-4o 0.85 0.92 0.90 5885 4715
Math GPT-4o-mini 0.72 0.88 0.87 5012 2855
Qwen3 0.86 0.95 0.95 5385 11049
Deepseek-V3 0.60 0.91 0.88 1804 2985
Llama3 0.64 0.80 0.77 4834 3749
Claude 3.7 0.59 0.57 0.79 2527 4226
Gemini 2.5 Pro 0.55 0.55 0.70 2882 2422
GPT-4o 0.46 0.48 0.58 3473 3529
Coding GPT-4o-mini 0.33 0.41 0.63 1170 2139
Qwen3 0.52 0.60 0.58 1176 4498
Deepseek-V3 0.33 0.36 0.44 1195 1443
Llama3 0.21 0.22 0.24 1498 1463
Claude 3.7 0.34 0.73 0.73 5212 3527
Gemini 2.5 Pro 0.47 0.68 0.65 6245 4540
GPT-4o 0.27 0.31 0.44 5589 4440
General GPT-4o-mini 0.13 0.29 0.33 4879 1071
Qwen3 0.28 0.42 0.40 4842 12125
Deepseek-V3 0.25 0.56 0.50 5169 1503
Llama3 0.22 0.35 0.28 2199 2012

C Experimental Procedure Formatting

MCP-Pool
"mcp_pool": [
{
"name": "FireCrawl",
"description": "A Model Context Protocol (MCP) server implementation that integrates with Firecrawl for
web scraping capabilities.",
"tools": [
{
"tool_name": "firecrawl_search",
"tool_description": "Search the web and optionally extract content from search results.",
"inputs": [
{
"name": "query",
"type": "string",
"required": true,
"description": "your search query"
}
]
}
],
"run_config": [
{
"command": "npx -y firecrawl-mcp",
"env": {
"FIRECRAWL_API_KEY": "your key"
},
"port": your port
}
]
}

16
 Eval-Result

{
"unique_id": "*****",
"question": "Given a integer n(>0), make a pile of n levels of ...",
"ground_truth": "def make_a_pile(n):\n
return [n + 2*i for i in range(n)]",
"prediction": "def make_a_pile(n):\n
pile = []\n for i in range(n):\n
if n % 2 == 0:\n [Link](n + 2*i)\n
else:\n [Link](n + 2*i)\n
return pile",
"success": true,
"tool_usage": {
"tool_calls": [
{"name": "read_file", "arguments": "{\"path\":\[Link]\"}"},
{"name": "write_file", "arguments": "{\"path\":\[Link]\"}"}
],
"total_tool_count": 2,
"tool_names": ["read_file", "write_file"]
},
"token_usage": {
"prompt_tokens": 820,
"completion_tokens": 610,
"total_tokens": 1430
}
}

D Others
Evaluation dimensions’ detailed introduction: Result Accuracy (RA). Result Accuracy is a funda-
mental metric measuring a model’s task completion success, defined as:
Nsuccess
RA = (1)
Ntotal
where Nsuccess represents the number of tasks successfully completed by the model, and Ntotal
represents the total number of tasks requiring tool use. The criterion for successful completion is
that the model’s final answer matches the standard answer, allowing for minor expression variants.
Unlike other dimensions, RA focuses exclusively on the task’s final result rather than process quality.
While simple, this metric is indispensable as a baseline measurement standard because, in practical
applications, the correctness of the final result is typically users’ primary concern, regardless of the
process. Although RA is a fundamental measurement standard, it alone cannot reflect the model’s
efficiency or robustness in tool utilization processes.
Dynamic Tool Selection Rate (DTSR). DTSR evaluates the accuracy of each call round in the
model’s tool utilization process, calculated as:
PNtotal
Ri
DT SR = 1 − i=1 (2)
Ntotal
where Ri represents the proportion of erroneous tool calls in i-th task. Unlike RA, which focuses
solely on final results, DTSR reflects the quality of model behavior throughout the entire interaction
process; even if the final result is correct, erroneous calls during the process will reduce the DTSR
score. This metric is particularly important for evaluating model performance in complex multi-step
tasks, as it reveals the model’s ability to maintain long-term tool usage consistency.
First Error Position (FEP). FEP measures the position of a model’s first error in the workflow,
assessing overall task resilience:
PNtotal
Di
F EP = i=1 (3)
Ntotal

17
where Di represents the proportional depth of the first erroneous tool call in the i-th task. For
successful tasks, Di = 1, indicating an error-free entire process; for failed tasks, higher Di values
indicate later error occurrence, suggesting the model can correctly execute longer tool call sequences.
Unlike DTSR, which focuses on overall error rates, FEP concentrates on the timing of errors,
particularly important for evaluating model stability in long-sequence reasoning. In complex tasks,
early errors are typically more fatal than later ones, and FEP effectively distinguishes this difference.
Computational Resource Efficiency (CRE). CRE measures the model’s efficient utilization of
computational resources during task execution, calculated based on token consumption:
Tb
CRE = (4)
Ta
where Tb represents the average token consumption of a benchmark model completing similar tasks,
and Ta represents the actual token consumption of the evaluated model. CRE values greater than
1 indicate higher efficiency than the benchmark, while values less than 1 indicate lower efficiency.
This metric is particularly important because, in production environments, computational efficiency
directly affects operational costs and response times.
Response Time Efficiency (RTE). RTE measures time efficiency from user input to final response:
tbaseline
RT E = (5)
tactual
where tbaseline represents the average response time of a benchmark model completing similar tasks,
and tactual represents the actual time for the model to generate a complete response. Similar to
CRE, RTE values greater than 1 indicate faster performance than the benchmark, while values less
than 1 indicate longer response times. In practical applications, response time directly affects user
experience, especially in scenarios requiring real-time interaction.
To intuitively display model performance distribution across the five dimensions, we normalize
the metrics to the [0, 1] interval and generate radar chart representations. Our primary goal is not
to provide a singular ranking but to reveal different models’ strengths and weaknesses through
multidimensional visualization, guiding selection for specific application scenarios. This multidi-
mensional evaluation framework allows MCP-R ADAR to transcend simple binary “task completed
or not” assessments, deeply understanding model behavioral characteristics, efficiency performance,
and resource consumption in tool usage processes, providing fine-grained feedback for developers
and researchers to optimize model and tool design.

18