S KILL RL: Evolving Agents via Recursive Skill-Augmented

Reinforcement Learning

Peng Xia 1 * Jianwen Chen 1 * Hanyang Wang 1 2 * Jiaqi Liu 1 Kaide Zeng 1 Yu Wang 3 Siwei Han 1
Yiyang Zhou 1 Xujiang Zhao 4 Haifeng Chen 4 Zeyu Zheng 5 Cihang Xie 6 Huaxiu Yao 1

Abstract
Memory Skills
Large Language Model (LLM) agents have shown
arXiv:2602.08234v1 [[Link]] 9 Feb 2026

Base Expert
Environment Trajectory
Model
stunning results in complex tasks, yet they often Discard Evolve
(a)
operate in isolation, failing to learn from past Higher Performance

experiences. Existing memory-based methods Faster Convergence

primarily store raw trajectories, which are often

redundant and noise-heavy. This prevents agents
from extracting high-level, reusable behavioral
patterns that are essential for generalization. In
this paper, we propose S KILL RL, a framework
that bridges the gap between raw experience and
(b)
policy improvement through automatic skill dis-
covery and recursive evolution. Our approach Figure 1. (a) Overview of the S KILL RL pipeline. Unlike previ-
introduces an experience-based distillation mech- ous methods (gray dashed lines) that store raw trajectories and
discard failures, S KILL RL employs an experience-based distilla-
anism to build a hierarchical skill library S KILL - tion mechanism to transform diverse experiences into structured
BANK, an adaptive retrieval strategy for general skills. (b) Performance on ALFWorld validation set (Shridhar
and task-specific heuristics, and a recursive evolu- et al.). S KILL RL achieves faster convergence and superior success
tion mechanism that allows the skill library to rates compared to vanilla GRPO and memory-augmented RL.
co-evolve with the agent’s policy during rein-
forcement learning. These innovations signifi- by interacting with complex environments through natural
cantly reduce the token footprint while enhancing language. Despite these advances, each task execution re-
reasoning utility. Experimental results on ALF- mains largely episodic. Current LLM agents operate in iso-
World, WebShop and seven search-augmented lation, unable to learn from past successes or failures (Zhang
tasks demonstrate that S KILL RL achieves state- et al., 2025b), which significantly hinders their evolution.
of-the-art performance, outperforming strong Consequently, a fundamental challenge remains: how can
baselines over 15.3% and maintaining robustness agents efficiently learn from experience and transfer that
as task complexity increases. Code is available at knowledge to other tasks?
this [Link] The existing memory-based methods for LLM agents pri-
marily involve saving raw trajectories directly into external
1. Introduction databases during the sampling process to serve as references
Large language model (LLM) agents (Yao et al., 2022b; for similar future tasks (Shinn et al., 2023; Zhao et al., 2024).
Shinn et al., 2023) have demonstrated remarkable capa- While intuitive, these raw trajectories are often lengthy and
bilities across various sophisticated tasks, such as web contain significant redundancy and noise (Chhikara et al.,
navigation (Google, 2025; OpenAI, 2025c) and deep re- 2025), making it difficult for the model to extract critical in-
search (OpenAI, 2025b; Google, 2024; Team et al., 2025), formation. Recent work has attempted to compress trajecto-
ries and update the memory bank via online training (Zhang
1
UNC-Chapel Hill 2 University of Chicago 3 University of et al., 2025b; 2026), improving memory efficiency. How-
California San Diego 4 NEC Labs America 5 University of Cal- ever, these methods merely mimic past solutions and they
ifornia Berkeley 6 University of California Santa Cruz. Cor- fail to distill core principles or adapt the agent’s internal
respondence to: Peng Xia <pxia@[Link]>, Huaxiu Yao
<huaxiu@[Link]>. policy to leverage memory for guided decision-making. As
depicted in the dashed flow of Figure 1(a), such approaches
Preprint. February 10, 2026. often struggle with the trade-off between information den-

1
 S KILL RL: Evolving Agents via Recursive Skill-Augmented Reinforcement Learning

sity and noise, leading to sub-optimal performance or even a state ot ∈ O, selects an action at ∈ A, and receives
degradation as shown in Figure 1(b). a reward rt and next observation ot+1 . A trajectory τ =
(o0 , a0 , r0 , . . . , oT , aT , rT ) captures one episode of interac-
We argue that these approaches miss a crucial insight: effec-
tion. Tasks are specified by natural language descriptions d.
tive experience transfer requires abstraction. Human experts
An LLM-based agent parameterized by θ implements a pol-
do not memorize every action in every situation; instead,
icy πθ (at |o≤t , d, c) where c represents additional context
they develop skills (Anthropic, 2024), compact and reusable
(e.g., skills, demonstrations). Our goal is to learn hP a policy
strategies that capture the essence of how to accomplish T
i
t
specific subtasks. Inspired by this observation, we propose that maximizes expected return maxθ Eτ ∼πθ t=0 γ rt
S KILL RL, a framework that bridges the gap between raw subject to context length constraints |c| ≤ Lmax .
experience and efficient policy improvement through auto-
Group Relative Policy Optimization (GRPO).
matic skill discovery and recursive skill evolution.
GRPO (Shao et al., 2024) is a reinforcement learn-
S KILL RL first introduces an experience-based skill distil- ing method that avoids training a critic by using intra-group
lation mechanism, which gathers diverse trajectories from relative rewards to optimize the policy. For each query x,
environment rollouts and applies differential processing: the model samples G responses {y (1) , . . . , y (G) }, which are
successful episodes are preserved as demonstrations, while scored to obtain rewards {R1 , . . . , RG }. GRPO computes
failed ones are synthesized into concise failure lessons to normalized advantages and updates the policy with a
mitigate context noise. Secondly, we transform these ex- PPO-style clipped objective (Schulman et al., 2017):
periences into a hierarchical skill library S KILL BANK, dif- " G
ferentiating between general skills for universal strategic 1 X 
JGRPO (θ) = Ex,{yi } min ri Ai ,
guidance and task-specific skills for task-level heuristics. G i=1
(1)
This abstraction allows the agent to adaptively retrieve rele- 
#
vant skills during decision-making, significantly reducing clip(ri , 1 − ϵ, 1 + ϵ)Ai − βDKL (πθ ∥πref ) ,
the token footprint while enhancing reasoning utility. Lastly,
S KILL RL incorporates a recursive skill evolution mecha- where ri = πθ (yi |x)
is the importance ratio, Ai =
πold (yi |x)
nism during reinforcement learning (RL), where the skill Ri −mean({Rj }G
j=1 )
library is treated as a dynamic component rather than a static std({Rj }G
is the normalized advantage, ϵ, β are hyper-
j=1 )
knowledge source. By analyzing failure modes after each parameters, and πold is the policy before the current update.
validation epoch to generate new skills or refine existing
ones, our approach ensures the skill library and the agent’s
policy co-evolve, maintaining robustness as task complex- 3. S KILL RL
ity increases. As demonstrated in Figure 1(b), S KILL RL
achieves substantially faster convergence and higher asymp- In this section, as illustrated in Figure 2, we propose
totic performance. S KILL RL, a framework designed to bridge the gap between
raw interaction experience and policy improvement through
The primary contribution is S KILL RL, a framework that en- automatic skill discovery and recursive evolution. S KILL RL
ables LLM agents to bridge the gap between raw experience consists of three core components. First, we develop an
and policy improvement through automatic skill discovery experience-based skill distillation mechanism to transform
and recursive evolution. By distilling redundant trajecto- redundant trajectories into concise, actionable knowledge.
ries into a hierarchical S KILL BANK, our method abstracts Second, we organize these distilled experiences into a hierar-
general and task-specific skills to guide decision-making chical skill library S, enabling efficient retrieval of general
efficiently. Furthermore, we introduce a recursive evolution and task-specific expertise. Lastly, we introduce a recursive
mechanism that ensures the skill library and agent policy co- skill evolution mechanism that leverages RL to dynamically
evolve during reinforcement learning. Empirical results on refine the skill library in tandem with the agent’s policy. We
ALFWorld, WebShop, and seven search-augmented bench- detail these components as follows:
marks demonstrate that S KILL RL achieves state-of-the-art
performance with 15.3% improvements, significantly out- 3.1. Experience-based Skill Distillation
performing current memory-based agent-tuning baselines Raw trajectories τ collected from environment interactions
in both task success and reasoning utility. are verbose, containing exploratory actions, backtracking,
and redundant steps that obscure the critical decisions lead-
2. Preliminaries ing to success or failure. To transform these experiences
into actionable knowledge, we employ a teacher model MT
LLM Agents. We consider an agent operating in an interac- to distill trajectories into compact, reusable skills.
tive environment E. At each timestep t, the agent observes
Specifically, we first deploy a base LLM agent πbase in the

2
 S KILL RL: Evolving Agents via Recursive Skill-Augmented Reinforcement Learning

Successful Episodes
a1 a2 ⋯ an
Lessons from Failure
Base Model Environment Trajectory Memory

Take what you see

Cold Start Skills
Prioritize exploring
General Skills unvisited nodes

For cleaning tasks,

Evolved Model Environment SFT Model Task-Specific Skills follow this process:
Locate -> Obtain ->
Clean the pool ->
Update SkillBank Create New or Refine Place this item

Figure 2. Overview of the S KILL RL framework. We collect trajectories using a base model, distill them into a hierarchical skill library,
perform cold-start SFT to enable skill utilization, and then conduct RL training with dynamic skill evolution based on validation failures.

target environment E to collect diverse trajectories. Unlike ciples applicable across all task types within an environment.
prior approaches that retain only successful episodes, we de- These typically include exploration strategies (e.g., system-
liberately preserve both successful trajectories T + = {τi : atic search patterns, prioritizing unvisited locations), state
r(τi ) = 1} and failed trajectories T − = {τi : r(τi ) = 0}, management principles (e.g., verifying preconditions be-
where r(τ ) denotes the binary task success indicator. Failed fore actions), and goal-tracking heuristics (e.g., maintaining
trajectories reveal failure modes and boundary conditions, progress counters, terminating only upon verified comple-
i.e., information difficult to infer from successes alone. tion). General skills provide foundational guidance that
transfers across different task categories. 2) Task-Specific
We apply differential processing based on trajectory out-
Skills Sk encode specialized knowledge for task category
comes. For successful trajectories τ + ∈ T + , we extract the
k. These capture domain-specific action sequences, task-
strategic patterns that led to task completion:
particular preconditions and constraints, common failure
s+ = MT (τ + , d). (2) modes unique to the task type, and optimized procedures
The teacher model identifies critical decision points, the that exploit task structure. By organizing trajectories by task
reasoning behind correct actions, and generalizable patterns type during collection, we enable extraction of fine-grained,
that transfer beyond the specific task instance. category-specific strategies that complement the broader
general skills.
For failed trajectories τ − ∈ T − , direct inclusion in context SK
is infeasible due to their length and noise. Instead, we The complete skill library S KILL BANK is Sg ∪ k=1 Sk .
synthesize concise failure lessons: Each skill s ∈ S KILL BANK is structured with: a concise
name (e.g., systematic exploration), a principle describing
s− = MT (τ − , d). (3) the strategy, and when to apply conditions specifying ap-
The analysis identifies: (1) the point of failure, (2) the flawed plicability. This format enables efficient retrieval while
reasoning or action, (3) what should have been done, and providing clear guidance for application.
(4) general principles to prevent similar failures. This trans-
forms verbose failed episodes into counterfactuals. Skill Retrieval. At inference, given a task description
d, the agent retrieves relevant skills to augment its con-
3.2. Hierarchical Skill Library (S KILL BANK) text. General skills Sg are always included as foundational
Construction guidance. Task-specific skills are retrieved via semantic
Following the design principles of Agent Skills (Anthropic, similarity:
2024), we organize the distilled knowledge into a hierarchi-
cal skill library S KILL BANK that enables efficient retrieval Sret = TopK ({s ∈ Sk : sim(ed , es ) > δ}, K) , (4)
of relevant expertise during decision-making.
where ed , es are embeddings of the task description and skill
Skill Organization. We structure S KILL BANK into two respectively, δ is a similarity threshold, and K controls the
levels: 1) General Skills Sg capture universal strategic prin- number of retrieved skills. The policy then conditions on

3
 S KILL RL: Evolving Agents via Recursive Skill-Augmented Reinforcement Learning

Algorithm 1 S KILL RL: Recursive Skill-Augmented RL (SFT) stage (Ouyang et al., 2022), where the teacher model
Require: Base model πbase , teacher MT , environment E MT generates N skill-augmented reasoning traces DSFT =
Ensure: Trained policy πθ∗ , evolved skill library S KILL BANK∗ {(di , Si , τi∗ )}N
i=1 demonstrating how to retrieve, interpret,
1: ▷ Experience-based Skill Distillation and apply skills during decision-making. The base model is
2: T + , T − ← Rollout(πbase , E) then fine-tuned on these demonstrations:
3: for all τ + ∈ T + do
4: s+ ← MT (τ + )
5: end for
6: for all τ − ∈ T − do θsft = arg min LCE (DSFT ; θ), (6)
θ
7: s− ← MT (τ − )
8: end for
9: ▷ Hierarchical Skill Library Construction where LCE denotes the cross-entropy loss. The resulting
10: Sg ← general skills from distilled experiences
11: for all task type k do model πθsft serves as both the starting point for RL training
12: Sk ← task-specific skills for category k and the reference policy πref for KL regularization.
13: end for
14: S KILL BANK ← Sg ∪ k Sk
S Recursive Skill Evolution. A static skill library cannot an-
15: ▷ Recursive Skill Evolution via RL ticipate all scenarios the agent will encounter. As the policy
16: // Cold-start initialization improves and explores new state regions, it faces situations
17: DSFT ← MT (E, S KILL BANK) where existing skills provide insufficient guidance. We in-
18: θ ← SFT(πbase , DSFT ); πref ← πθ troduce recursive skill evolution to address this limitation.
19: // RL with recursive evolution
20: for epoch = 1 to N do The process begins with an initial skill library containing
21: for all task d do baseline task-action principles.
22: Sret ← Retrieve(d, S KILL BANK)
After each validation epoch, we monitor the success rate
23: Sample {τ (i) }Gi=1 ∼ πθ (·|d, Sg , Sret )
Compute {Ri }G
Acc(C) for each task category C. To ensure targeted
24: i=1 and update θ via GRPO
25: end for growth, the evolution is triggered only for categories where
−
26: if validation epoch then Acc(C) < δ. We then collect failed trajectories Tval = {τj :
− M
27: Tval ← failed validation trajectories r(τj ) = 0}j=1 using a diversity-aware stratified sampling
−
28: Snew ← MT (Tval , S KILL BANK) strategy: trajectories are grouped by category, prioritized by
29: S KILL BANK ← S KILL BANK ∪ Snew the severity of failure (negative rewards), and selected via
30: end if
31: end for round-robin sampling to maintain categorical entropy. Then
32: return πθ , S KILL BANK we will analyze these samples to identify gaps:

−
the retrieved skills: Snew = MT (Tval , S KILL BANK). (7)

at ∼ πθ (at |o≤t , d, Sg , Sret ). (5)

The teacher model is prompted to: (1) identify failure pat-
Notably, skill distillation achieves 10–20× token compres- terns not addressed by current skills, (2) propose new skills
sion compared to raw trajectories while enhancing rather to cover these gaps, and (3) suggest refinements to existing
than degrading the utility of the original experience. This skills that proved ineffective. The library is then updated:
compression allows the agent to leverage rich experiential S KILL BANK ← S KILL BANK ∪ Snew .
knowledge within limited context windows.
This creates a virtuous cycle: as the agent improves, it en-
3.3. Recursive Skill Evolution counters new challenges, which drive skill library expansion,
which enables further improvement.
A static skill library cannot anticipate all scenarios the agent
RL-based Policy Optimization. We optimize the skill-
will encounter. As the policy improves and explores new
augmented policy using GRPO. For each task with descrip-
state regions, it faces situations where existing skills provide
tion d, the agent first retrieves relevant skills and then sam-
insufficient guidance. We introduce recursive skill evolution
ples G complete trajectories {τ (1) , . . . , τ (G) } from the cur-
during reinforcement learning to address this limitation,
rent policy πθ . Each trajectory τ (i) receives a binary reward
enabling the skill library and agent policy to co-evolve.
Ri = r(τ (i) ) ∈ {0, 1} indicating task successfulness. The
Cold-Start Initialization. Before RL training, we address normalized advantage for each trajectory is computed as:
a critical challenge: the base agent has not learned how
to effectively utilize skills. Simply providing skills to an
unchanged model yields limited benefit (Guo et al., 2025). Ri − mean({Rj }G j=1 )
Ai = . (8)
We therefore perform a cold-start supervised fine-tuning std({Rj }G
j=1 )

4
 S KILL RL: Evolving Agents via Recursive Skill-Augmented Reinforcement Learning

The policy is updated according to: and GRPO (Shao et al., 2024) that optimize policies via
" G
advantage estimation over trajectory groups. Finally, we
1 X 
compare against memory-augmented RL-based methods,
J (θ) = Ed,{τ (i) } min ρi Ai ,
G i=1 such as EvolveR (Wu et al., 2025), MemRL (Zhang et al.,
# (9) 2026), and the combination of Mem0+GRPO and Sim-

clip(ρi , 1 − ϵ, 1 + ϵ)Ai − βDKL (πθ ∥πref ) , pleMem (Liu et al., 2026)+GRPO, which integrate per-
sistent memory mechanisms directly into the reinforce-
ment learning optimization process to handle long-term
π (τ (i) |d,S ,S )
where ρi = πoldθ (τ (i) |d,Sgg ,Sretret ) is the importance ratio com- dependencies. For search-augmented QA, we compare
puted over the skill-augmented context. The KL penalty S KILL RL with R1-Instruct, Search-o1 (Li et al., 2025),
anchored to πref = πθsft ensures that RL optimization pre- Search-R1 (Jin et al., 2025), ZeroSearch (Sun et al., 2025),
serves the learned skill utilization capabilities while improv- and StepSearch (Zheng et al., 2025).
ing task performance. The complete training procedure is Implementation Details. We use Qwen2.5-7B-
summarized in Algorithm 1. Instruct (Bai et al., 2023) as our base model and
OpenAI o3 (OpenAI, 2025a) as the teacher model for skill
4. Experiments distillation and SFT data generation. For RL training, we
use GRPO with learning rate 1 × 10−6 , batch size 16, group
We evaluate S KILL RL on nine challenging benchmarks size 8, and 4 gradient accumulation steps. We set K = 6 for
for LLM agents: ALFWorld, WebShop, and seven search- task-specific skill retrieval and δ = 0.4 for the collection
augmented QA tasks. Our experiments address the follow- of failed trajectories. For more detailed information on
ing questions: 1) How does S KILL RL compare to state- training hyperparameters, please see Appendix B.1.
of-the-art methods? 2) What is the contribution of each
component? 3) How does the skill library evolve during 4.2. Main Results
training? 4) Does skills accelerate model convergence?
Comparison with Baselines. We compare S KILL RL with
4.1. Experimental Setup baseline methods across two benchmarks as shown in Ta-
ble 1. Our method consistently outperforms all baselines,
Environments. ALFWorld (Shridhar et al.) is a text-based with key observations as follows:
game aligned with the ALFRED embodied AI benchmark.
Agents must complete household tasks by navigating and 1) Significant Gains over Prompt-based Methods. S KILL RL
interacting with objects through text commands. WebShop achieves a 89.9% success rate on ALFWorld and 72.7% on
(Yao et al., 2022a) simulates web shopping. Agents navigate WebShop, outperforming the best prompt-based baselines
a realistic web interface to find and purchase products match- by a large margin. This gap suggests that while in-context
ing user specifications. In addition, we also evaluate the learning can leverage past experiences, it often fails to distill
performance of S KILL RL on search-augmented QA tasks, actionable knowledge from verbose trajectories or funda-
including single-hop QA datasets (NQ (Kwiatkowski et al., mentally adapt the agent’s policy.
2019), TriviaQA (Joshi et al., 2017), and PopQA (Mallen 2) Superiority over Vanilla RL. RL training brings sub-
et al., 2023)) and multi-hop QA datasets (HotpotQA (Yang stantial gains, yet S KILL RL consistently surpasses stan-
et al., 2018), 2Wiki (Ho et al., 2020), MuSiQue (Trivedi dard RL baselines. Compared to PPO, RLOO, and GRPO,
et al., 2022), and Bamboogle (Press et al., 2023)). S KILL RL achieves the best overall performance. Notably,
Baselines. We compare S KILL RL against four categories since S KILL RL utilizes GRPO as its base optimizer, the
of competitive methods. First, we include closed-source 12.3% absolute improvement over GRPO on ALFWorld
LLMs, specifically GPT-4o (OpenAI, 2024) and Gemini- (from 77.6% to 89.9%) is directly attributable to our skill-
2.5-Pro (Comanici et al., 2025), which represent the state-of- augmentation mechanism rather than algorithmic variance.
the-art in general-purpose reasoning and instruction follow- In complex subtasks like Cool and Pick2, S KILL RL out-
ing. Second, we evaluate prompt-based agentic or memory- performs GRPO by 23.0% and 22.8% respectively, proving
based methods, including ReAct (Yao et al., 2022b) and Re- that structured skill priors effectively accelerate and enhance
flexion (Shinn et al., 2023), which rely on in-context prompt- policy learning in sparse-reward environments.
ing for multi-step reasoning, as well as Mem0 (Chhikara 3) Advantage over Memory-Augmented RL. S KILL RL sub-
et al., 2025), ExpeL (Zhao et al., 2024), and MemP (Fang stantially outperforms existing memory-augmented RL
et al., 2025), which utilize external memory or experience frameworks, which differ in how they manage and update
pools to guide behavior without parameter updates. Third, experience. MemRL, which uses RL solely to update its
we consider RL-based methods, including group-based on- memory bank while keeping the policy frozen, fails to adapt
line RL algorithms such as RLOO (Ahmadian et al., 2024)

5
 S KILL RL: Evolving Agents via Recursive Skill-Augmented Reinforcement Learning

Table 1. Performance on ALFWorld and WebShop. For ALFWorld, we report the average success rate (%) for each subtask as well as
the overall result. For WebShop, we report both the average score and the average success rate (%). ∗ denotes the results replicated
from (Feng et al., 2025). The best results and second best results are highlighted in red and blue , respectively.

ALFWorld WebShop
Method
Pick Look Clean Heat Cool Pick2 All Score Succ.
Closed-source LLMs
GPT-4o 75.3 60.8 31.2 56.7 21.6 49.8 48.0 31.8 23.7
Gemini-2.5-Pro 92.8 63.3 62.1 69.0 26.6 58.7 60.3 42.5 35.9
Qwen2.5-7B-Instruct
Qwen2.5 33.4 21.6 19.3 6.90 2.80 3.20 14.8 26.4 7.80
Prompt-based Agentic or Memory-based Methods
ReAct∗ 48.5 35.4 34.3 13.2 18.2 17.6 31.2 46.2 19.5
Reflexion∗ 62.0 41.6 44.9 30.9 36.3 23.8 42.7 58.1 28.8
Mem0 54.0 55.0 26.9 36.4 20.8 7.69 33.6 23.9 2.00
ExpeL 21.0 67.0 55.0 52.0 71.0 6.00 46.3 30.9 11.2
MemP 54.3 38.5 48.1 56.2 32.0 16.7 41.4 25.3 6.40
SimpleMem 64.5 33.3 20.0 12.5 33.3 3.84 29.7 33.2 8.59
RL-based Methods
RLOO∗ 87.6 78.2 87.3 81.3 71.9 48.9 75.5 80.3 65.7
GRPO∗ 90.8 66.1 89.3 74.7 72.5 64.7 77.6 79.3 66.1
Memory-Augmented RL-based Methods
MemRL 62.8 38.5 22.2 12.5 8.00 0.00 21.4 29.5 9.20
EvolveR 64.9 33.3 46.4 13.3 33.3 33.3 43.8 42.5 17.6
Mem0+GRPO 78.1 54.8 56.1 31.0 65.0 26.9 54.7 58.1 37.5
SimpleMem+GRPO 89.5 36.3 60.0 50.0 64.9 26.3 62.5 67.8 46.9
S KILL RL 97.9 71.4 90.0 90.0 95.5 87.5 89.9 85.2 72.7

to complex environments, yielding only 21.4% on ALF- Table 2, S KILL RL achieves a state-of-the-art average score
World. EvolveR, which jointly updates the policy and mem- of 47.1%, significantly outperforming Search-R1 (38.5%)
ory bank, shows improvement (43.8%) but remains limited abd EvolveR (43.1%). Key observations include: 1) Su-
by its reliance on rough trajectory storage. To provide a perior multi-hop Reasoning: S KILL RL excels in complex
more competitive baseline, we implemented Mem0+GRPO, tasks like Bamboogle, surpassing EvolveR by 19.4%. This
which combines a state-of-the-art prompt-based memory demonstrates that hierarchical skills effectively guide multi-
mechanism with an optimized policy model. While this step information synthesis. 2) Strong generalization: De-
hybrid approach improves performance to 54.7% on ALF- spite being trained on limited datasets (NQ, HotpotQA),
World and 37.5% on WebShop, it still trails S KILL RL by S KILL RL maintains competitive performance on OOD tasks
a wide margin (about 35.2% absolute success rate gap). like TriviaQA and 2Wiki, confirming that distilled search
These results validate our core hypothesis: effective expe- strategies are task-agnostic.
rience transfer requires high-level skill abstraction and a
co-evolving library rather than simple trajectory compres- 4.3. Analysis
sion or prompt-based memory retrieval.
In this section, we provide detailed analysis of each mod-
Comparison with Closed-Source Models. Remarkably, ule’s effectiveness and the skill evolution dynamics.
S KILL RL with Qwen2.5-7B-Instruct significantly outper-
forms much larger closed-source models, as shown in Ta- Ablation Studies. We conduct ablation experiments to eval-
ble 1. On ALFWorld, our method exceeds GPT-4o (OpenAI, uate each component’s contribution, with results in Table 3.
2024) by 41.9% and Gemini-2.5-Pro (Comanici et al., 2025) According to the results: (1) Removing hierarchical struc-
by 29.6%. This demonstrates that effective skill learning ture (i.e., task-specific skills only) decreases performance
can compensate for model scale, enabling smaller open- by 13.1% on ALFWorld and 11.3% on WebShop, indicating
source models to achieve superior task performance through universal strategic principles provide essential foundational
structured experiential knowledge. guidance. (2) Replacing the skill library with raw trajec-
tories causes the largest degradation (up to 25%), which
Performance on Search-Augmented QA. As shown in

6
 S KILL RL: Evolving Agents via Recursive Skill-Augmented Reinforcement Learning
† ⋆
Table 2. Performance on search-augmented QA tasks. S KILL RL is trained on NQ and HotpotQA. and indicate in-domain and
out-of-domain datasets, respectively. ∗ denotes the results replicated from (Sun et al., 2025).

Single-Hop QA Multi-Hop QA
Method Avg.
NQ† TriviaQA⋆ PopQA⋆ HotpotQA† 2Wiki⋆ MuSiQue⋆ Bamboogle⋆
Qwen2.5-7B-Instruct
Qwen2.5∗ 11.6 35.6 1.20 16.4 22.2 4.80 14.4 15.2
CoT∗ 12.8 35.6 3.80 16.2 22.6 6.60 24.0 17.4
RAG∗ 27.4 58.2 17.8 25.8 23.2 9.40 16.8 25.5
Search-o1∗ 19.4 40.6 11.4 17.0 27.0 8.60 30.4 22.1
R1-Instruct 21.0 44.9 17.1 20.8 27.5 6.00 19.2 22.4
Search-R1 39.3 61.0 39.7 37.0 40.1 14.6 36.8 38.5
ZeroSearch 43.6 61.8 51.5 34.6 35.2 18.4 27.8 39.1
StepSearch - - - 38.6 36.6 22.6 40.0 -
EvolveR 43.5 63.4 44.6 38.2 42.0 15.6 54.4 43.1
S KILL RL 45.9 63.3 45.9 43.2 40.3 20.2 73.8 47.1

Table 3. Ablation study results. We report average success rate (%) Skill Library Evolution
on ALFWorld and WebShop. Total Skills Heat
120 General Cool
Pick Pick2
Method ALFWorld WebShop Look Mistakes
Clean 100
100 94
S KILL RL 89.9 72.7 88
83
Number of Skills
Skill Library Ablations 80 77
71
w/o Hierarchical Structure 76.8 61.4 66
w/o Skill Library (Raw Trajectories) 61.7 50.2 60
60 55
Training Pipeline Ablations
w/o Cold-Start SFT 65.2 46.5 40
w/o Dynamic Evolution 84.4 70.3
20

0 0 20 40 60 80 100 120 140 150

directly supports our motivation that abstraction is superior Training Steps
to memorization. Raw experiences introduce significant re-
Figure 3. Evolution of skill library size during RL training. Dy-
dundancy and noise that hinder effective knowledge transfer. namic skill evolution adds skills at validation checkpoints.
(3) Cold-start SFT proves critical (20% drop without it),
confirming that the base model requires an initial explicit
skills show a steadier increase (from 12 to 20). Notably, we
demonstration phase to learn how to adaptively retrieve and
observe a balanced expansion across various task categories,
utilize the abstracted skills before entering the RL stage.
ensuring the agent develops specialized expertise for each
(4) Dynamic evolution contributes a 5.5% improvement by
environment rollout. This overall expansion reflects the
ensuring the skill library is a dynamic component rather
agent’s increasing ability to refine its repertoire and tackle
than a static database. This co-evolution allows the agent to
diverse scenarios within specific task types.
iteratively refine its internal policy by addressing emergent
failure modes that were not covered by the initial skill set. Context Efficiency. To evaluate the impact of skill abstrac-
tion on inference overhead, we compare the average prompt
Per-Task Analysis on ALFWorld. Table 1 breaks down
length of S KILL RL with a memory-augmented baseline
ALFWorld performance by task type. The largest gains are
using raw trajectories (Qwen2.5-7B with Raw Memory) in
on PickTwo (+23%), Cool (+22%) and Heat (+15%), which
Figure 4. The results reveal that while the raw memory
are among the most challenging tasks requiring multi-step
approach suffers from a high and fluctuating token footprint
planning and state tracking. Task-specific skills are particu-
(averaging ∼1,450 tokens), S KILL RL maintains a signifi-
larly valuable here, capturing strategies like “when picking
cantly leaner prompt (averaging <1,300 tokens), achieving
two objects, verify the first is secured before searching for
approximately a 10.3% reduction in context length. This ef-
the second” that address common failure modes.
ficiency stems from our distillation mechanism, which com-
Skill Library Growth. Figure 3 shows how the skill library presses verbose environment interactions into high-density,
evolves during training. The initial skill library contains actionable skills. Notably, S KILL RL requires less context
55 skills (12 general, 43 task-specific). Through dynamic than the memory-based baseline to achieve superior per-
evolution, this grows to 100 skills by the end of training formance, demonstrating that skill abstraction effectively
(Step 150). The growth is predominantly driven by task- mitigates the context-bloat problem common in traditional
specific skills (increasing from 43 to 80), while general memory-based agents.

7
 S KILL RL: Evolving Agents via Recursive Skill-Augmented Reinforcement Learning

Figure 4. Comparison of prompt length (tokens) between raw mem- Figure 5. Success rate on ALFWorld validation set. The recursive
ory retrieval and our distilled skill abstraction. S KILL RL consis- skill evolution significantly accelerates convergence and enhances
tently reduces context overhead while maintaining reasoning utility. the overall performance ceiling.

Evolution Dynamics. Figure 5 illustrates the reinforce- ning during interaction, while Reflexion (Shinn et al., 2023)
ment learning training curves with and without the recur- introduces verbal reinforcement through self-reflection on
sive skill evolution mechanism. We observe that while past failures. Frameworks like AutoGen (Wu et al., 2024)
S KILL RL without evolution shows steady improvement, and CAMEL (Li et al., 2023) demonstrate general-purpose
S KILL RL with skill evolution exhibits a notably higher multi-agent capabilities, featuring automated orchestration
learning rate and superior asymptotic performance. Specifi- and diverse tool integration. While initial efforts focused on
cally, S KILL RL achieves a success rate of over 80% within constrained tasks like coding or basic arithmetic, these ap-
60 training steps, whereas the baseline requires approxi- proaches primarily rely on in-context learning (ICL) (Dong
mately 90 steps to reach a lower peak. This acceleration et al., 2024). However, these agents struggle to scale as tasks
in convergence suggests that the dynamic introduction of become more complex, as they treat every interaction as an
new skills and refinement of existing ones effectively pro- isolated event and must start each new task from scratch
vide the agent with timely strategic guidance to overcome without any prior knowledge.
local optima. Furthermore, the higher performance ceiling
Memory Mechanisms in Agents. To overcome the limita-
validates that the co-evolution of the skill library and the
tions of finite context windows and the inability of agents to
policy allows the agent to adapt to increasingly complex
learn from experience, external memory architectures have
task scenarios that static memory methods fail to resolve.
become a cornerstone of agent design (Hu et al., 2025;
Qualitative Analysis. To further investigate how S KILL RL Wang, 2025). Early systems primarily utilized a static
utilizes the learned knowledge, we visualize the reasoning RAG paradigm or stored raw trajectories as few-shot ex-
process on ALFWorld and WebShop in Figure 6. The case amples (Wang et al.; Chhikara et al., 2025; Zhang et al.,
studies demonstrate that our trained agent can effectively 2025a; Wang et al., 2024). However, raw trajectories are
retrieve and execute relevant skills from the S KILL BANK often token-heavy and contain significant redundancy and
to guide its decision-making. For instance, in the Web- noise, which can lead to performance degradation. Current
Shop task, the agent invokes general strategies like “Pri- research has moved toward self-improving memory, dis-
oritize Core Keywords” alongside task-specific heuristics tilling interactions into higher-level insights or procedural
“Focus Key Query” to ensure the product meets all constraints tips (Wang & Chen, 2025; Tang et al., 2025; Fang et al.,
within a limited budget. Similarly, in ALFWorld, the agent 2025; Zhao et al., 2024; Ouyang et al., 2025; Wei et al.,
coordinates hierarchical skills, i.e., using “Progressive Goal 2025). While some recent work explores updating memory
Decomposition” for high-level planning and “No Appliance banks via online training to improve efficiency (Zhang et al.,
Before Object” to avoid common logical pitfalls. This seam- 2025b; 2026), many existing methods still struggle to dis-
less integration of general and specific skills confirms that tinguish high-value experiences from noise or fail to distill
the agent does not merely memorize trajectories, but rather core principles that can guide internal decision-making.
develops a structured understanding of task logic, allowing
Evolution of Agentic Skills and Reinforcement Learning.
for more robust and efficient problem-solving.
The development of agentic skills (Anthropic, 2024), which
are compact, reusable strategies that capture the essence
5. Related Work of subtasks, is increasingly viewed through the lens of
Continual Learning (CL) and RL. Traditional CL (Parisi
LLM Agents. The emergence of capable LLMs has
et al., 2019) focuses on knowledge preservation in prede-
catalyzed rapid development in autonomous agent sys-
fined tasks, but self-evolving agents (Gao et al., 2025; Xia
tems (Wei et al., 2026). ReAct (Yao et al., 2022b) interleaves
et al., 2025; Liu et al., 2025) aim for active skill acquisition
reasoning and acting, enabling chain-of-thought style plan-
in open-ended environments (Fang et al., 2025; Wang et al.,

8
 S KILL RL: Evolving Agents via Recursive Skill-Augmented Reinforcement Learning

WebShop ALFWorld
Task: I need a women's long sleeve button-down shirt in navy blue, size large, Task: heat some egg and put it in countertop.
machine washable, price lower than $40.00
--- Retrieved Skills --- --- Retrieved Skills ---
Detected Task Type: apparel Detected Task Type: heat
[General Skills] … [Task-Specific Skills] [General Skills] … [Task-Specific Skills]
------ ------
Step 1 Step 1
Observation: WebShop homepage. Search bar available. Observation: You are in the middle of a room. Looking quickly around you…
<think>Following skill [gen_001] 'Prioritize Core Keywords' and [app_001] 'Focus Key <think>Per [gen_010] 'Progressive Goal Decomposition': sub-goals are … Per [hea_004]
Query': I should include product type + must-have attributes. My query should be …</think> 'No Appliance Before Object': I must NOT go to microwave first. Per [hea_001]
'Secure Exact Target First': find the egg. It's food — check fridge and
…
<action>search[women long sleeve button down shirt navy blue machine …]</action>
countertops.</think>
Step 9
Observation: Product page updated: Color = Navy Blue, Size = L. Price: $29.99
…
<action>go to countertop 1</action>
Step 8
<think>Applying skill [app_004] 'Check Variant Price': price for Navy Blue / L is $29.99,
Observation: You arrive at countertop 1
which is under the $40 budget. Also per [app_005] 'Confirm Hidden Specs': … $29.99 <
<think>Per [hea_005]: place the heated egg directly.</think>
$40. Per skill [gen_013] 'Purchase Decisively', I should buy now.</think>
<action>put egg 1 in/on countertop 1</action>
<action>click[Buy Now]</action>

Figure 6. Case studies of S KILL RL on WebShop and ALFWorld. The examples illustrate how the agent adaptively retrieves and integrates
General Skills and Task-Specific Skills within its reasoning process to achieve precise and efficient task execution.

2025). While RL is widely used to align LLMs (Schulman 2024.

et al., 2017; Ouyang et al., 2022), or improve reasoning
via rule-based verifiers (Shao et al., 2024), applying it to Anthropic. The claude 3 model family: Opus, son-
agentic skills remains challenging due to sparse rewards net, haiku, 2024. URL [Link]
and long horizons. Unlike previous memory-augmented RL com/news/claude-3-family.
which treats memory as a static or auxiliary source, recent
Bai, J., Bai, S., Chu, Y., Cui, Z., Dang, K., Deng, X., Fan,
trends suggest that the key to efficient experience transfer
Y., Ge, W., Han, Y., Huang, F., et al. Qwen technical
lies in abstraction (Wu et al., 2025). Our work builds on this
report. arXiv preprint arXiv:2309.16609, 2023.
by treating the skill library as a dynamic component that
co-evolves with the agent’s policy, utilizing RL to refine Chhikara, P., Khant, D., Aryan, S., Singh, T., and Yadav, D.
structured skills through recursive failure analysis. Mem0: Building production-ready ai agents with scalable
long-term memory. arXiv preprint arXiv:2504.19413,
6. Conclusion 2025.

We introduced S KILL RL, a framework for skill-augmented Comanici, G., Bieber, E., Schaekermann, M., Pasupat, I.,
reinforcement learning in LLM agents. By distilling raw Sachdeva, N., Dhillon, I., Blistein, M., Ram, O., Zhang,
trajectories into compact, reusable skills and enabling dy- D., Rosen, E., et al. Gemini 2.5: Pushing the frontier
namic skill evolution during training, S KILL RL achieves with advanced reasoning, multimodality, long context,
state-of-the-art performance on ALFWorld and WebShop and next generation agentic capabilities. arXiv preprint
while using substantially less context than memory-based arXiv:2507.06261, 2025.
approaches. Our work demonstrates that the abstraction
from experience to skill is a powerful principle for building Dong, Q., Li, L., Dai, D., Zheng, C., Ma, J., Li, R., Xia,
capable, sample-efficient agents. H., Xu, J., Wu, Z., Chang, B., et al. A survey on in-
context learning. In Proceedings of the 2024 conference
on empirical methods in natural language processing, pp.
Acknowledgement 1107–1128, 2024.
This work was partially supported by the Amazon Research Fang, R., Liang, Y., Wang, X., Wu, J., Qiao, S., Xie,
Award, the Cisco Faculty Research Award, NEC Laborato- P., Huang, F., Chen, H., and Zhang, N. Memp:
ries America Research Grant, and Coefficient Giving. Exploring agent procedural memory. arXiv preprint
arXiv:2508.06433, 2025.
References
Feng, L., Xue, Z., Liu, T., and An, B. Group-in-group
Ahmadian, A., Cremer, C., Gallé, M., Fadaee, M., Kreutzer, policy optimization for llm agent training. arXiv preprint
J., Pietquin, O., Üstün, A., and Hooker, S. Back to basics: arXiv:2505.10978, 2025.
Revisiting reinforce-style optimization for learning from
human feedback in llms. In Proceedings of the 62nd Gao, H.-a., Geng, J., Hua, W., Hu, M., Juan, X., Liu, H., Liu,
Annual Meeting of the Association for Computational S., Qiu, J., Qi, X., Wu, Y., et al. A survey of self-evolving
Linguistics (Volume 1: Long Papers), pp. 12248–12267, agents: On path to artificial super intelligence. arXiv
preprint arXiv:2507.21046, 2025.

9
 S KILL RL: Evolving Agents via Recursive Skill-Augmented Reinforcement Learning

Google. Try deep research and our new ex- Liu, J., Su, Y., Xia, P., Han, S., Zheng, Z., Xie, C., Ding, M.,
perimental model in gemini, your ai assistant, and Yao, H. Simplemem: Efficient lifelong memory for
2024. URL [Link] llm agents. arXiv preprint arXiv:2601.02553, 2026.
gemini/google-gemini-deep-research/.
Mallen, A., Asai, A., Zhong, V., Das, R., Khashabi, D., and
Google. Introducing the gemini 2.5 computer Hajishirzi, H. When not to trust language models: Inves-
use model, 2025. URL [Link] tigating effectiveness of parametric and non-parametric
google/technology/google-deepmind/ memories. In Proceedings of the 61st Annual Meeting of
gemini-computer-use-model/. the Association for Computational Linguistics (Volume 1:
Long Papers), pp. 9802–9822, 2023.
Guo, D., Yang, D., Zhang, H., Song, J., Zhang, R., Xu, R.,
Zhu, Q., Ma, S., Wang, P., Bi, X., et al. Deepseek-r1: In- OpenAI. Gpt-4o system card, 2024. [Link]
centivizing reasoning capability in llms via reinforcement com/index/gpt-4o-system-card/.
learning. arXiv preprint arXiv:2501.12948, 2025. OpenAI. Introducing o3 and o4-mini,
2025a. [Link]
Ho, X., Nguyen, A.-K. D., Sugawara, S., and Aizawa, A. introducing-o3-and-o4-mini/.
Constructing a multi-hop qa dataset for comprehensive
evaluation of reasoning steps. In Proceedings of the 28th OpenAI. Openai deep research system card,
International Conference on Computational Linguistics, 2025b. URL [Link]
pp. 6609–6625, 2020. introducing-deep-research/.

Hu, Y., Liu, S., Yue, Y., Zhang, G., Liu, B., Zhu, F., Lin, J., OpenAI. Openai computer-using agent, 2025c.
Guo, H., Dou, S., Xi, Z., et al. Memory in the age of ai URL [Link]
agents. arXiv preprint arXiv:2512.13564, 2025. computer-using-agent/.
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C.,
Jin, B., Zeng, H., Yue, Z., Yoon, J., Arik, S., Wang, D.,
Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A.,
Zamani, H., and Han, J. Search-r1: Training llms to
et al. Training language models to follow instructions
reason and leverage search engines with reinforcement
with human feedback. Advances in neural information
learning. arXiv preprint arXiv:2503.09516, 2025.
processing systems, 35:27730–27744, 2022.
Joshi, M., Choi, E., Weld, D. S., and Zettlemoyer, L. Trivi- Ouyang, S., Yan, J., Hsu, I., Chen, Y., Jiang, K., Wang,
aqa: A large scale distantly supervised challenge dataset Z., Han, R., Le, L. T., Daruki, S., Tang, X., et al. Rea-
for reading comprehension. In Proceedings of the 55th soningbank: Scaling agent self-evolving with reasoning
Annual Meeting of the Association for Computational memory. arXiv preprint arXiv:2509.25140, 2025.
Linguistics (Volume 1: Long Papers), pp. 1601–1611,
2017. Parisi, G. I., Kemker, R., Part, J. L., Kanan, C., and Wermter,
S. Continual lifelong learning with neural networks: A
Kwiatkowski, T., Palomaki, J., Redfield, O., Collins, M., review. Neural networks, 113:54–71, 2019.
Parikh, A., Alberti, C., Epstein, D., Polosukhin, I., Devlin,
J., Lee, K., et al. Natural questions: a benchmark for ques- Press, O., Zhang, M., Min, S., Schmidt, L., Smith, N. A.,
tion answering research. Transactions of the Association and Lewis, M. Measuring and narrowing the composi-
for Computational Linguistics, 7:453–466, 2019. tionality gap in language models. In Findings of the As-
sociation for Computational Linguistics: EMNLP 2023,
Li, G., Hammoud, H., Itani, H., Khizbullin, D., and Ghanem, pp. 5687–5711, 2023.
B. Camel: Communicative agents for” mind” exploration Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and
of large language model society. Advances in Neural Klimov, O. Proximal policy optimization algorithms.
Information Processing Systems, 36:51991–52008, 2023. arXiv preprint arXiv:1707.06347, 2017.
Li, X., Dong, G., Jin, J., Zhang, Y., Zhou, Y., Zhu, Y., Zhang, Shao, Z., Wang, P., Zhu, Q., Xu, R., Song, J., Bi, X., Zhang,
P., and Dou, Z. Search-o1: Agentic search-enhanced H., Zhang, M., Li, Y., Wu, Y., et al. Deepseekmath: Push-
large reasoning models. arXiv preprint arXiv:2501.05366, ing the limits of mathematical reasoning in open language
2025. models. arXiv preprint arXiv:2402.03300, 2024.
Liu, J., Xiong, K., Xia, P., Zhou, Y., Ji, H., Feng, L., Han, Shinn, N., Cassano, F., Gopinath, A., Narasimhan, K., and
S., Ding, M., and Yao, H. Agent0-vl: Exploring self- Yao, S. Reflexion: Language agents with verbal rein-
evolving agent for tool-integrated vision-language reason- forcement learning. Advances in Neural Information
ing. arXiv preprint arXiv:2511.19900, 2025. Processing Systems, 36:8634–8652, 2023.

10
 S KILL RL: Evolving Agents via Recursive Skill-Augmented Reinforcement Learning

Shridhar, M., Yuan, X., Cote, M.-A., Bisk, Y., Trischler, A., Wu, Q., Bansal, G., Zhang, J., Wu, Y., Li, B., Zhu, E., Jiang,
and Hausknecht, M. Alfworld: Aligning text and embod- L., Zhang, X., Zhang, S., Liu, J., et al. Autogen: Enabling
ied environments for interactive learning. In International next-gen llm applications via multi-agent conversations.
Conference on Learning Representations. In First Conference on Language Modeling, 2024.

Sun, H., Qiao, Z., Guo, J., Fan, X., Hou, Y., Jiang, Y., Xie, Wu, R., Wang, X., Mei, J., Cai, P., Fu, D., Yang, C., Wen, L.,
P., Zhang, Y., Huang, F., and Zhou, J. Zerosearch: In- Yang, X., Shen, Y., Wang, Y., et al. Evolver: Self-evolving
centivize the search capability of llms without searching. llm agents through an experience-driven lifecycle. arXiv
arXiv preprint arXiv:2505.04588, 2025. preprint arXiv:2510.16079, 2025.

Tang, X., Qin, T., Peng, T., Zhou, Z., Shao, D., Du, T., Wei, Xia, P., Zeng, K., Liu, J., Qin, C., Wu, F., Zhou, Y.,
X., Xia, P., Wu, F., Zhu, H., et al. Agent kb: Leveraging Xiong, C., and Yao, H. Agent0: Unleashing self-evolving
cross-domain experience for agentic problem solving. agents from zero data via tool-integrated reasoning. arXiv
arXiv preprint arXiv:2507.06229, 2025. preprint arXiv:2511.16043, 2025.

Team, T. D., Li, B., Zhang, B., Zhang, D., Huang, F., Li, G., Yang, Z., Qi, P., Zhang, S., Bengio, Y., Cohen, W., Salakhut-
Chen, G., Yin, H., Wu, J., Zhou, J., et al. Tongyi deepre- dinov, R., and Manning, C. D. Hotpotqa: A dataset for
search technical report. arXiv preprint arXiv:2510.24701, diverse, explainable multi-hop question answering. In
2025. Proceedings of the 2018 conference on empirical methods
in natural language processing, pp. 2369–2380, 2018.
Trivedi, H., Balasubramanian, N., Khot, T., and Sabharwal,
A. Musique: Multihop questions via single-hop ques- Yao, S., Chen, H., Yang, J., and Narasimhan, K. Web-
tion composition. Transactions of the Association for shop: Towards scalable real-world web interaction with
Computational Linguistics, 10:539–554, 2022. grounded language agents. Advances in Neural Informa-
tion Processing Systems, 35:20744–20757, 2022a.
Wang, G., Xie, Y., Jiang, Y., Mandlekar, A., Xiao, C., Zhu,
Y., Fan, L., and Anandkumar, A. Voyager: An open-ended Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan,
embodied agent with large language models. Transac- K. R., and Cao, Y. React: Synergizing reasoning and
tions on Machine Learning Research. acting in language models. In The eleventh international
conference on learning representations, 2022b.
Wang, Y. From Static Parameters to Updatable Memory:
Enabling Large Language Model Agents to Remember, Zhang, G., Fu, M., Wan, G., Yu, M., Wang, K., and Yan, S.
Adapt, and Learn. PhD thesis, University of California, G-memory: Tracing hierarchical memory for multi-agent
San Diego, 2025. systems. arXiv preprint arXiv:2506.07398, 2025a.

Wang, Y. and Chen, X. Mirix: Multi-agent memory system Zhang, G., Ren, H., Zhan, C., Zhou, Z., Wang, J., Zhu, H.,
for llm-based agents. arXiv preprint arXiv:2507.07957, Zhou, W., and Yan, S. Memevolve: Meta-evolution of
2025. agent memory systems. arXiv preprint arXiv:2512.18746,
2025b.
Wang, Y., Takanobu, R., Liang, Z., Mao, Y., Hu, Y.,
McAuley, J., and Wu, X. Mem-{\alpha}: Learning Zhang, S., Wang, J., Zhou, R., Liao, J., Feng, Y., Zhang,
memory construction via reinforcement learning. arXiv W., Wen, Y., Li, Z., Xiong, F., Qi, Y., et al. Memrl:
preprint arXiv:2509.25911, 2025. Self-evolving agents via runtime reinforcement learning
on episodic memory. arXiv preprint arXiv:2601.03192,
Wang, Z. Z., Mao, J., Fried, D., and Neubig, G. Agent 2026.
workflow memory. arXiv preprint arXiv:2409.07429,
2024. Zhao, A., Huang, D., Xu, Q., Lin, M., Liu, Y.-J., and Huang,
G. Expel: Llm agents are experiential learners. In Pro-
Wei, T., Sachdeva, N., Coleman, B., He, Z., Bei, Y., Ning, ceedings of the AAAI Conference on Artificial Intelli-
X., Ai, M., Li, Y., He, J., Chi, E. H., et al. Evo-memory: gence, volume 38, pp. 19632–19642, 2024.
Benchmarking llm agent test-time learning with self-
evolving memory. arXiv preprint arXiv:2511.20857, Zheng, X., An, K., Wang, Z., Wang, Y., and Wu, Y.
2025. Stepsearch: Igniting llms search ability via step-wise
proximal policy optimization. In Proceedings of the 2025
Wei, T., Li, T.-W., Liu, Z., Ning, X., Yang, Z., Zou, J., Zeng, Conference on Empirical Methods in Natural Language
Z., Qiu, R., Lin, X., Fu, D., et al. Agentic reasoning for Processing, pp. 21816–21841, 2025.
large language models. arXiv preprint arXiv:2601.12538,
2026.

11
 S KILL RL: Evolving Agents via Recursive Skill-Augmented Reinforcement Learning

Appendix

A. Prompts
In this section, we provide the full prompt templates used throughout the different phases of our framework. These templates
are designed to ensure consistent agent behavior and structured data generation across various environments.

A.1. Agent Execution Prompts

The following prompts are used during the online inference phase. These templates provide the agent with the current task
description, a history of previous interactions, and a set of retrieved skills (experiences) to guide its decision-making process.
The prompts explicitly enforce a Chain-of-Thought (CoT) reasoning step before action selection.

Prompt A.1: ALFWorld Agent Execution with Skills

System Prompt:
You are an expert agent operating in the ALFRED Embodied Environment. Your task is to: {task description}

## Retrieved Relevant Experience

{retrieved memories}

## Current Progress
Prior to this step, you have already taken {step count} step(s). Below are the most recent {history length} observations
and the corresponding actions you took: {action history}
You are now at step {current step} and your current observation is: {current observation}
Your admissible actions of the current situation are: [{admissible actions}].

Now it’s your turn to take an action. You should first reason step-by-step about the current situation. This reasoning process MUST
be enclosed within <think> </think> tags. Once you’ve finished your reasoning, you should choose an admissible action for
current step and present it within <action> </action> tags.

Prompt A.2: WebShop Agent Execution with Skills

System Prompt:
You are an expert autonomous agent operating in the WebShop e-commerce environment. Your task is to: {task description}.

## Retrieved Relevant Experience

{retrieved memories}

## Current Progress
Prior to this step, you have already taken {step count} step(s). Below are the most recent {history length} observations
and the corresponding actions you took: {action history}
You are now at step {current step} and your current observation is: {current observation}
Your admissible actions of the current situation are: [ {available actions} ].

Now it’s your turn to take one action for the current step. You should first reason step-by-step about the current situation, then think
carefully which admissible action best advances the shopping goal. This reasoning process MUST be enclosed within <think>
</think> tags. Once you’ve finished your reasoning, you should choose an admissible action for current step and present it within
<action> </action> tags.

A.2. Skill Generation and Distillation Prompts

These prompts are utilized during the skill discovery and library initialization phases. They guide a high-capability teacher
model to analyze interaction trajectories, identify failure modes, and distill reusable, actionable skills into a structured JSON
format.

12
 S KILL RL: Evolving Agents via Recursive Skill-Augmented Reinforcement Learning

Prompt B.1: Dynamic Skill Discovery from Failures

Analyze these failed {env description} agent trajectories and suggest NEW skills to add.

FAILED TRAJECTORIES: {failure examples}

EXISTING SKILL TITLES: {existing titles}

Generate 1-3 NEW actionable skills that would help avoid these failures. Each skill must have: skill id, title (3-5 words),
principle (1-2 sentences), when to apply. The skill id should be unique and follow the pattern: ”dyn 001”, ”dyn 002”,
etc.

Return ONLY a JSON array of skills, no other text.

Prompt B.2: Initial Skill Distillation (ALFWorld)

You are an expert at distilling agent behavior patterns into concise, actionable skills. Analyze these successful and failed trajectories
from an embodied AI agent operating in household environments (ALFWorld).

SUCCESSFUL TRAJECTORIES: {success patterns}

FAILED TRAJECTORIES: {failure patterns}

Generate 8-12 GENERAL SKILLS that apply across ALL task types. These should be: 1. Concise; 2. Actionable; 3. Transferable;
4. Failure-aware. Focus on: Navigation, object manipulation, state tracking, error recovery, and container interaction rules.

Return ONLY the JSON array, no other text.

Prompt B.3: Initial Skill Distillation (WebShop)

You are an expert at distilling agent behavior patterns into concise, actionable skills. Analyze these successful and failed trajectories
from an AI agent operating in an online shopping environment (WebShop).

SUCCESSFUL TRAJECTORIES: {success patterns}

FAILED TRAJECTORIES: {failure patterns}

Generate 10-15 GENERAL SKILLS. Focus on: Search query formulation, product selection heuristics, option configuration (size,
color, etc.), constraint verification, navigation patterns, and price handling.

Return ONLY the JSON array, no other text.

A.3. Cold-start Trajectory Generation Prompts

To bridge the gap between a base model and the target performance, we use the following prompts to generate high-quality
synthetic trajectories for Supervised Fine-Tuning (SFT). These prompts instruct the teacher model to solve tasks while
explicitly demonstrating the application of specific skills, thereby providing a clear learning signal for the student model.

Prompt C.1: Synthetic Trajectory Generation (ALFWorld)

You are an expert agent in the ALFRED embodied environment. You will be given a task and relevant skills to apply. Your goal is to
generate a successful trajectory that demonstrates proper use of these skills.

You should generate a step-by-step trajectory that:

1. Uses the provided skills appropriately;
2. Takes realistic actions in the environment;
3. Completes the task successfully;
4. Demonstrates good planning and systematic exploration.

For each step, you should:

• Think through the current situation using <think></think> tags.
• Choose an appropriate action using <action></action> tags.
• The action should be a simple command like ”go to cabinet 1”, ”open drawer 2”, ”take apple 1”, ”put apple 1 in/on countertop 1”.

Generate a complete trajectory from start to finish. Stop when the task is complete.

13
 S KILL RL: Evolving Agents via Recursive Skill-Augmented Reinforcement Learning

Prompt C.2: Synthetic Trajectory Generation (WebShop)

You are an expert shopping agent in the WebShop e-commerce environment. You will be given a shopping task and relevant skills to
apply. Your goal is to generate a successful trajectory that demonstrates proper use of these skills.

You should generate a step-by-step trajectory that:

1. Uses the provided skills appropriately;
2. Takes realistic actions in the WebShop environment;
3. Successfully finds and purchases the requested product;
4. Demonstrates good search strategies and product evaluation.

For each step, you should:

• Think through the current situation using <think></think> tags.
• Choose an appropriate action using <action></action> tags.
• Actions can be: search[query], click[element], or buy now.

Generate a complete trajectory from start to finish. Stop when the purchase is complete.

B. Additional Experimental Details

B.1. Hyperparameters

Table 4. Hyperparameters for S KILL RL.

Hyperparameter Value
Cold-Start SFT
Learning rate 1 × 10−4
Batch size 16
Epochs 3
SFT examples 7,500 (AlfWorld) / 2,400 (WebShop)
RL Training
Learning rate 1 × 10−6
Batch size 64
KL loss Coef 0.01
Invalid Action Penalty Coef 0.1
Max Prompt Length 6,000
Max Response Length 1,024
Epoch 150
Skill Retrieval
Top-K retrieval 6
Validation interval 5 Steps
Update Threshold δ 0.4
Max failures analyzed 10 (SR < 0.4) / 5 (SR > 0.4)
Max new skills per evolution 3

B.2. Compute Resources

All experiments were conducted on a cluster with 8 NVIDIA H100 80GB GPUs. Training times:

• Trajectory collection: 3 hours

• Skill distillation: 0.5 hours

• Cold-start SFT: 2 hour

• RL training: 24 hours

14
 S KILL RL: Evolving Agents via Recursive Skill-Augmented Reinforcement Learning

Table 5. Example distilled skills from S KILL BANK for ALFWorld (Shridhar et al.). This table summarizes general patterns and application
logic derived from raw trajectories.

ID Skill Title Principle (Actionable Pattern) When to Apply

General Exploration & Acquisition Skills
gen 001 Systematic Exploration Search every plausible surface or container exactly once before Anytime the goal count is not
revisiting; prioritize unseen locations. met and unexplored areas re-
main.
gen 002 Immediate Acquisition As soon as a required object becomes visible and reachable, Upon first visual confirmation
take it immediately. of a goal-relevant object.
gen 003 Destination First Policy After picking up a goal object, navigate directly to the known Holding any goal object while
target receptacle and place it. target location is identified.
State-Changing & Spatial Relation Skills
gen 005 Use State-Changing Acquire the object, then immediately use the nearest suitable After picking up an object re-
Tools Early appliance (heat/cool/clean) before placement. quiring temperature or cleanli-
ness change.
gen 006 Establish Spatial Rela- First locate the reference object, adjust its state if needed, then Tasks containing prepositions
tions search or place in the specified region. like “under”, “inside”, or “on”.
Reliability & Error Recovery
gen 014 Loop Escape Trigger If the last 3–5 actions do not change the state, switch to an After several consecutive no-
untried search branch or action type. progress observations.
gen 015 Pre-Action Sanity Confirm prerequisites (hand free, capacity, power) before exe- Right before issuing any com-
Check cuting manipulative commands. mand that could legally fail.

Table 6. Common Agent Failures and Mitigation Strategies for ALFWorld.

ID Failure Description Root Cause (Why it happens) Mitigation (How to avoid)

err 001 Redundant Revisit Lacks explicit memory of explored areas; strat- Maintain an exploration map; prioritize unvis-
egy degenerates into local loops. ited candidates.
err 006 Skipping State Changes Conflates object presence with goal satisfac- Integrate state precondition checks into the
tion; omits cleanliness/temp checks. planner before placement.

Total wall-clock time: approximately 30 hours per experiment.

C. Illustration of Skill Library

In this section, we provide some example catalog of distilled skills and error taxonomies for both the ALFWorld and
WebShop environments. Tables 5 and 7 detail the general skills distilled for embodied manipulation and web-based
shopping, respectively, highlighting the actionable principles required for systematic exploration and constraint satisfaction.
Furthermore, we provide a structured analysis of failure cases in Table 6 and Table 8, which categorizes common mistakes,
ranging from spatial reasoning loops in ALFWorld to price-shift oversights in WebShop, alongside their root causes and
proposed mitigation strategies.

15
 S KILL RL: Evolving Agents via Recursive Skill-Augmented Reinforcement Learning

Table 7. Example distilled skills for WebShop Navigation (Yao et al., 2022a). These skills represent the strategic patterns used by the
agent to handle large-scale product search and constraint satisfaction.

ID Skill Title Principle (Actionable Pattern) When to Apply

Search & Query Engineering
gen 001 Prioritize Core Keywords Include product type, 1-2 functional attributes, and hard Before issuing the first search or re-
constraints; omit secondary descriptors. fining over-specific queries.
gen 002 Iterative Refinement Adjust keywords or apply site filters instead of repeat- When results are irrelevant or repeat
ing the same failed query. despite multiple searches.
Product Evaluation & Verification
gen 003 Scan Before You Click Read titles, thumbnails, and prices in results to ensure On search results pages when choos-
plausibility before opening a link. ing the next product to inspect.
gen 004 Verify Early, Abort Fast Immediately check category, attributes, and price on Within the first observation on every
the product page; leave if any constraint is violated. product detail page.
gen 006 Confirm Hidden Attributes Open Description/Features sections to ensure non- When constraints are not evident
visible specs (e.g., material) meet constraints. from the title or variant list.
Configuration & Transaction
gen 005 Set Mandatory Variants Always select required options (size, color, etc.) before After confirming product match but
evaluating price or purchasing. before any purchase action.
gen 007 Check Variant Pricing For price ranges, select the exact variant combination Whenever price changes with vari-
to verify the specific price is within budget. ant selection or shows as a range.
gen 013 Purchase Decisively Execute ’Buy Now’ immediately once all constraints After validating every constraint on
and prices are confirmed on a variant. the current product variant.

Table 8. Common Failures in Web-based Shopping Tasks.

ID Failure Description Root Cause Mitigation Strategy

err 001 Missing Constraints in Omits size or price caps, leading to over- Assemble full requirement list first; ensure
Query whelming or irrelevant result sets. every hard constraint is in the query string.
err 004 Price Shift Oversight Fails to notice price changes after selecting a Re-read the price element after every option
specific size or color variant. change before proceeding to checkout.
err 005 Premature Purchase Clicks “Buy Now” without setting mandatory Validate that every required dropdown/radio
variants, leading to errors or wrong items. option is explicitly selected before buying.
err 009 Ignoring Stock Status Attempts to purchase out-of-stock items by Verify that the ’Add to Cart’ button is enabled
ignoring disabled buttons or stock labels. and no ’Out of Stock’ message is present post-
selection.
err 011 Sponsored Link Distraction Clicks loosely matched ads, diverting the Implement ad-label detection; prioritize or-
workflow from organic, suitable products. ganic listings for higher constraint reliability.

16
 S KILL RL: Evolving Agents via Recursive Skill-Augmented Reinforcement Learning

D. Additional Cases
Case 1: WebShop Footwear (Functional Constraints)

Task: I want a pair of men’s black slip-resistant work shoes, size 10, rubber sole, price lower than $50.00.

Skills [foo 002] Verify features in description; [foo 004] Confirm price after variant selection.
Avoid [err 001] Omitting price cap; [err 003] Selecting wrong product category.

Agent Trajectory with Skill-Guided Reasoning

Step 1 Obs: WebShop homepage.

Think: Per [gen 001], I’ll include key constraints (type, function, color, price) in the query.
Action: search[men black slip resistant work shoes rubber sole under 50]
Step 2 Obs: Results: [1] Chef Shoes $38.99; [2] Kitchen Shoes $42.50; [3] Steel Toe Boots $55.99.
Think: [gen 003] Item [1] is under budget and matches function. Item [3] is over budget (skip).
Action: click[Item 1]
Step 3 Obs: Product page. Price: $35.99-$45.99. Colors: Black, White. Sizes: 7-12.
Think: [foo 002] Description confirms rubber sole. Now selecting Black and Size 10 [foo 003].
Action: click[Black] → click[10]
Step 5 Obs: Selected: Black, Size 10. Price updated to $38.99.
Think: [foo 004] $38.99 ¡ $50. All constraints (color, size, function, price) are satisfied.
Action: click[Buy Now]

Result: SUCCESS — Purchased at $38.99. Key Skills: [foo 002] (Verification), [foo 004] (Price Check).

Case 2: Electronics (Speaker with Abort-and-Retry)

Task: I need a portable bluetooth speaker, waterproof, black color, price lower than $30.00.

Skills [ele 001] Constraint-rich search; [ele 004] Bail on mismatch fast; [gen 002] Iterative refinement.
Avoid [err 002] Overly specific variant names in search; [ele 006] Avoid click loops.

Agent Trajectory: Handling Missing Constraints

Step 1 Obs: WebShop homepage.

Think: [ele 001] Packing all mandatory attributes: portable, bluetooth, waterproof, black, ¡$30.
Action: search[portable bluetooth speaker waterproof black under 30]
Step 2 Obs: Results: [1] Waterproof Speaker $28.99; [2] Mini LED Speaker $19.99; [3] 20W Speaker $35.50.
Think: [ele 005] Item [1] matches most keywords and price.
Action: click[Item 1]
Step 3 Obs: Product page (Item 1). Colors: Blue, Red, Green, Army Green. Black unavailable.
Think: [gen 004] Verify Early, Abort Fast: No black option! [ele 004] Bail immediately to find another
candidate.
Action: click[Back to Search]
Step 4 Obs: Back on search results.
Think: [gen 002] Iterative refinement. Item [2] $19.99 is the next best candidate.
Action: click[Item 2]
Step 5 Obs: Product page (Item 2). Colors: Black, Blue, Red. Specs: IPX5 Waterproof.
Think: [ele 002] Black is available! IPX5 confirms waterproof. Selecting Black.
Action: click[Black]
Step 6 Obs: Black selected. Final price: $19.99.
Think: [ele 003] $19.99 ¡ $30. All specs (black, portable, waterproof) verified.
Action: click[Buy Now]

Result: SUCCESS — Efficiently handled attribute mismatch via early exit. Key Skills Applied: [gen 004] (Abort Fast), [ele 004]
(Bail on Mismatch).

17
 S KILL RL: Evolving Agents via Recursive Skill-Augmented Reinforcement Learning

Case 1: ALFWorld Clean Task (Lettuce → Sink → Countertop)

Task: Clean some lettuce and put it in countertop.

Core Skills [cle 001] Phase-ordered plan (Locate → Clean → Place); [cle 006] Use location priors (Fridge for food).
Mistakes Avoided [err 002] Skipping closed containers; [err 003] Ignoring visible target objects.

Step 1-3 Obs: Fridge 1 is closed. Action: open fridge 1 → take lettuce 1
Reasoning: Per [cle 006], food is likely in the fridge. [gen 002] Grab lettuce immediately on sight.
Step 4-6 Obs: Arrive at sinkbasin 1. Action: clean lettuce 1 with sinkbasin 1
Reasoning: Per [cle 001] Phase 2: bring to water source. [cle 003] Go straight to sink.
Step 7 Obs: Arrive at countertop 1. Action: put lettuce 1 in/on countertop 1
Reasoning: Per [gen 003] Destination First: place cleaned object immediately to finish.
Result: SUCCESS (7 Steps) — Skills used: [gen 010] (Decomposition), [cle 003] (Sink First).

Case 2: ALFWorld Heat Task (Egg → Microwave → Countertop)

Task: Heat some egg and put it in countertop.

Core Skills [hea 001] Secure exact target first; [hea 003] Open-Place-Heat sequence; [hea 004] No appliance before object.

Step 1-3 Obs: Countertop 1 (no egg) → Countertop 2 (egg found). Action: take egg 1
Reasoning: [hea 004] Avoid microwave until object is held. [hea 002] Systematic search of surfaces.
Step 4-6 Obs: Microwave 1 is closed. Action: open microwave 1 → heat egg 1
Reasoning: [hea 003] Correct sequence: open door first, then initiate state change.
Step 7-8 Obs: Egg is heated. Action: go to countertop 1 → put egg 1
Reasoning: [hea 005] Direct Post-Heat Placement: move to target without detours.
Result: SUCCESS (8 Steps) — Skills used: [hea 001] (Target First), [hea 004] (Sequence Control).

Case 3: ALFWorld Examine Task (Pencil ⊕ Desklamp → Success)

Task: Look at pencil under the desklamp.

Core Skills [loo 001] Seek lamp surfaces (Desk/Sidetable); [loo 002] Switch lamp on; [exa 001] Pair objects early
(Target & Tool co-location).
Mistakes Avoided [err 003] Ignoring reachable targets; [loo 005] Preventing action loops (Toggle only once).

Agent Trajectory: High-Efficiency Co-location Handling

Step 1 Obs: Mid-room. Visible: desk 1/2, drawer 1-3, sidetable 1, etc.
Reasoning: [loo 001] Desklamps usually sit on desks. Prioritizing desk 1 to find both pencil and lamp in one
go.
Action: go to desk 1
Step 2 Obs: On desk 1: alarmclock 1, desklamp 1, pencil 1.
Reasoning: [exa 001] Successful co-location! Both objects are present. [exa 002] Pick up target first before
interacting with the tool.
Action: take pencil 1 from desk 1
Step 3 Obs: Holding pencil 1.
Reasoning: [loo 002] Toggle the lamp state to satisfy the ”under light” condition. [exa 004] Single toggle rule
ensures no repeated actions.
Action: use desklamp 1

Result: SUCCESS (Optimal 3 Steps) — Demonstrates superior spatial reasoning and zero-waste navigation. Key Skills: [exa 001]
(Object Pairing), [loo 001] (Spatial Priors).

18