Merge, Ensemble, and Cooperate!

A Survey on Collaborative Strategies in

the Era of Large Language Models
Jinliang Lu1,2∗ , Ziliang Pang1∗ , Min Xiao1,2∗ , Yaochen Zhu3 * , Rui Xia3 , Jiajun Zhang1,2,4 †
1
Institute of Automation, Chinese Academy of Sciences, Beijing, China
2
School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China
3
Nanjing University of Science and Technology, Nanjing, China
4
Wuhan AI Research, Wuhan, China
{lujinliang2019, [Link]}@[Link],
{yczhu, rxia}@[Link], {jjzhang, [Link]}@[Link]

Abstract
GPT-4
The remarkable success of Large Language LLaMA-Chat
WebGPT
Models (LLMs) has ushered natural language ChatGPT
CodeLLaMA
CodeX
processing (NLP) research into a new era. De- Gemini - 1/1.5
LLaMA-Guard
arXiv:2407.06089v1 [[Link]] 8 Jul 2024

GPT-3
spite their diverse capabilities, LLMs trained on PaLM-1/2 LLaMA - 1/2/3
ERNIE

different corpora exhibit varying strengths and Gemma

Flan-T5
Qwen - 1/1.5/2
weaknesses, leading to challenges in maximiz- DeepSeek - Coder - 1/2
Qwen-Multilingual
ing their overall efficiency and versatility. To DeepSeek - Chat- 1/2 Qwen-Audio
Mistral
address these challenges, recent studies have ex- Baichuan Codestral
plored collaborative strategies for LLMs. This
paper provides a comprehensive overview of
this emerging research area, highlighting the
motivation behind such collaborations. Specifi-
cally, we categorize collaborative strategies into Figure 1: Recently, numerous large language models
three primary approaches: Merging, Ensemble, have been released, each with its own unique strengths.
and Cooperation. Merging involves integrating This diversity has fueled research into collaboration
multiple LLMs in the parameter space. En- between these models.
semble combines the outputs of various LLMs.
Cooperation leverages different LLMs to allow
full play to their diverse capabilities for spe- emerged as one of the most rapidly developing
cific tasks. We provide in-depth introductions and promising directions in artificial intelligence.
to these methods from different perspectives These models have significantly transformed the
and discuss their potential applications. Addi-
tionally, we outline future research directions,
paradigm of natural language processing (NLP)
hoping this work will catalyze further studies (Min et al., 2023a; Chang et al., 2024; Zhao et al.,
on LLM collaborations and paving the way for 2023) and influenced other areas (Wu et al., 2023a;
advanced NLP applications. Zhang et al., 2024a). This impressive revolution
has inspired numerous universities, institutes, and
1 Introduction companies to pre-train and release their own LLMs.
"Many hands make light work." Currently, over 74,000 pre-trained models are avail-
—– John Heywood able on the HuggingFace model hub1 . As shown in
Figure 1, these models, trained with diverse data,
Human beings have long understood the power of
architectures, and methodologies, possess unique
collaboration. When individuals pool their diverse
capabilities: some are proficient in multilingual
skills and efforts, they can achieve far more than
tasks (Le Scao et al., 2023; Lin et al., 2022), others
they could alone. This principle of collective effort
specialize in domains like medicine (Yang et al.,
has found new relevance in the realm of machine
2024b) or finance (Wu et al., 2023b), some are
learning (Dietterich, 2000; Panait and Luke, 2005;
adept at processing long-context windows (Chen
Sagi and Rokach, 2018), significantly boosting the
et al., 2023e,f), while others are fine-tuned for
development of artificial intelligence.
better alignment with human interaction (Ouyang
In recent years, large language models (LLMs)
et al., 2022). However, no single model consis-
(Brown et al., 2020; Chowdhery et al., 2023) have
tently outperforms all others across tasks (Jiang
* Equal Contribution
† 1
Corresponding author [Link]

1
et al., 2023a). This variability motivates research guage models (Vaswani et al., 2017; Radford et al.,
into the collaboration between various LLMs to 2018) to substantial more than billions of param-
unlock their combined potential, akin to creating a eters, exemplified by models like GPT-3 (Brown
Hexagon Warrior. et al., 2020), PaLM (Chowdhery et al., 2023; Anil
Despite progress in LLM collaboration research, et al., 2023), LLaMA (Touvron et al., 2023a,b).
the relationships and context among the proposed These models are typically considered as Large
methods remain unclear. This survey aims to fill Language Models (LLMs) due to their massive
that gap by categorizing collaboration techniques amount of parameters (Zhao et al., 2023). This
into three main approaches: Merging, Ensemble, subsection discusses the architecture and scaling of
and Cooperation. Specifically, Merging and En- LLMs, their training objectives, and the emergent
semble methods for LLMs are derived from tradi- abilities they exhibit.
tional fusion techniques commonly explored in ma-
chine learning (Li et al., 2023a). These methods are Architecture and Scaling Similar to pre-trained
tailored to be more suitable for LLMs, effectively language models (PLMs) (Radford et al., 2018; De-
leveraging the collaborative advantages of diverse vlin et al., 2019), LLMs primarily adopt the Trans-
LLMs. Merging involves integrating the parame- former architecture (Vaswani et al., 2017) as their
ters of multiple LLMs into a single, unified model, backbone, consisting of stacked multi-head atten-
requiring that the parameters are compatible within tion and feed-forward layers. Unlike PLMs, most
a linear space. In contrast, Ensemble focuses on currently released LLMs are built upon decoder-
combining the outputs generated by various LLMs only architectures for training efficiency and few-
to produce coherent results, with less emphasis on shot capabilities. This approach also shows po-
the parameters of the individual models. Coopera- tential when the number of parameters increases
tion extends beyond merging and ensemble. This (Zhang et al., 2022). Recent studies have investi-
survey concentrates on cooperative methods that gated the quantitative relationship between model
harness the diverse strengths of LLMs to achieve capacity, the amount of training data, and model
specific objectives. In general, these techniques size, known as the scaling law (Kaplan et al., 2020;
expand the methodologies for model collaboration, Hoffmann et al., 2022).
holding significant research importance for LLMs.
The structure of this work is organized as follows. Training Objectives In the previous studies
We begin by providing the background of LLMs about PLMs, various language modeling tasks are
and defining collaboration techniques for LLMs proposed. For example, masked language model-
in Section 2. Next, we introduce three key cate- ing for BERT (Devlin et al., 2019), De-noising lan-
gories: Merging in Section 3, Ensemble in Section guage modeling for BART (Lewis et al., 2020) and
4, and Cooperation in Section 5. Each category T5 (Raffel et al., 2020). However, current LLMs
of methods is thoroughly classified and described typically utilize the standard causal language mod-
in detail, offering a clear understanding of their eling as their training objective, which aims to pre-
respective frameworks and applications. Finally, dict the next token based on the preceding tokens
we offer a comprehensive discussion in Section 6, in a sequence. This training objective is well-suited
highlighting challenges and future directions for for decoder-only architectures.
research. Beyond the pre-training objective, recent studies
In summary, this study aims to comprehensively have aimed to model human preferences to better
explore the strategies and methodologies for col- align LLMs with human expectations. For exam-
laborative efforts among LLMs. We aspire for this ple, the well-known InstructGPT (Ouyang et al.,
survey to enrich understanding of LLM collabora- 2022) introduces reinforcement learning from hu-
tion strategies and to inspire future research. man feedback (RLHF), which uses preference re-
wards as an additional training objective. Although
2 Background RLHF is effective at making LLMs more helpful to
users, it inevitably incurs an alignment tax, which
2.1 Large Language Models refers to performance degradation after RLHF. Re-
Language modeling has always been a cornerstone cent research has explored various techniques to
of natural language processing (NLP). Recently, mitigate alignment tax issues (Lin et al., 2023; Lu
plenty of studies scale up of Transformer-based lan- et al., 2024b; Fu et al., 2024b).

2
 Who discovered the law of gravity?
Guys, I need help!

Isaac Newton

Final Answer

Isaac Newton Isaac Newton

How can we assist you?
Newton
Contrast?
Checking?
Compression?

(a) LLM Merging (b) LLM Ensemble (c) LLM Cooperation

Figure 2: The illustration of different collaboration strategies, with each animal in the figures representing a different
LLM.

Emergent Abilities The fundamental capability where I refers to the given instruction for cur-
of language models is text generation, where tokens rent example x. The instruction tuning technique
are auto-regressively generated based on preceding (Sanh et al., 2021; Ouyang et al., 2022; Wei et al.,
tokens using greedy search or nucleus sampling 2022a) can enhance the generalization capabilities
(Holtzman et al., 2020a): of LLMs, enabling them to perform well with in-
structions on a variety of tasks, including unseen
yi ∼ p(yi |y<i ) (1) ones (Thoppilan et al., 2022).

Interestingly, LLMs can not only generate real- 2.2 Collaboration for LLMs
istic text but also perform specific tasks when pro- For previous task-dependent NLP models, collabo-
vided with task-specific prompts, without requiring ration strategies typically aimed to improve perfor-
fine-tuning on particular downstream tasks (Brown mance on specific tasks (Jia et al., 2023). Recently,
et al., 2020). This phenomenon is one of the most LLMs have revolutionized NLP by demonstrating
important differences between LLMs and previ- remarkable versatility across a wide range of tasks.
ous PLMs. Wei et al. (2022b) define the emergent This shift has also shifted the focus of collabora-
ability as “an ability that is not present in smaller tion strategies for LLMs toward enhancing versa-
models but is present in larger models.” Among tility and achieving more general objectives. Con-
these emergent abilities, in-context learning (ICL) sequently, some recently proposed collaboration
(Brown et al., 2020; Dong et al., 2022) and instruc- strategies have become more flexible and tailored
tion following are commonly used and significantly specifically for LLMs.
enhance the ability of LLMs to process various
tasks. The Necessity of LLM Collaboration Although
ICL helps LLMs understand tasks by using sev- almost all LLMs demonstrate strong versatility
eral task examples as demonstrations. When pro- across various tasks through in-context learning
vide these demonstrations as prompts, LLMs can and instruction following, different LLMs still have
automatically generate reasonable output for the distinct strengths and weaknesses (Jiang et al.,
given test example, which can be formalized as: 2023a).
Differences in training corpora and model ar-
p(y|x) = p(y|x, demonstration({(xi , yi )}ki=1 )) chitectures among various LLM families—such
(2) as LLaMA, GLM (Zeng et al., 2023), and QWen
(Bai et al., 2023)—result in significant variations
Instruction following ability are typically emerge in their capabilities. Even within the same fam-
in LLMs that have been fine-tuned on examples ily, fine-tuning on specific corpora (e.g., mathemat-
formatted with instructions on multiple tasks. The ics (Azerbayev et al., 2023), code (Roziere et al.,
generation process can be formalized as: 2023), or medical domains (Wu et al., 2024)) can
lead to noticeable performance differences. Effec-
p(y|x) = p(y|x, I) (3) tive collaboration among these LLMs can unlock

3
 Basic M-ROS
e.g. Soup (Wortsman et al., 2022), Rame et al.
Methodolo-
Merging for (2022), Wan et al. (2024b), Liu et al. (2024b)
gies (§3.1.1)
Relatively Optimal
Solution (M-
ROS) (§3.1) Adaptation to e.g. Wan et al. (2024b), Liu et al. (2024b), Kim
LLMs (§3.1.2) et al. (2024), Fu et al. (2024a), Lin et al. (2023)

Merging (§3) Methods based

e.g. Jin et al. (2022), Daheim
on Weighted
et al. (2023), Nathan et al. (2024)
Average (§3.2.1)

Merging for
Methods based
Enhancing Multi- e.g. Ilharco et al. (2023), Yadav et al. (2023), Yang
on Task Prop-
Task Capability et al. (2023), Zhou et al. (2024a), Yu et al. (2023)
erty (§3.2.2)
(M-MTC) (§3.2)

Methods based
on Incremental e.g. Tang et al. (2023), Yang et al. (2024a)
Training (§3.2.3)

Before Infer- e.g. Shnitzer et al. (2023), Lu et al. (2023),

ence (§4.1.1) Srivatsa et al. (2024), Hu et al. (2024)

LLM Ensemble During Infer- e.g. Hoang et al. (2023), Li et al. (2024c),
Methodology (§4.1) ence (§4.1.2) Xu et al. (2024b), Huang et al. (2024c)

After Infer- e.g. Chen et al. (2023d), Madaan et al.

Ensemble (§4)
LLM Collaboration ence (§4.1.3) (2023), Yue et al. (2024), Jiang et al. (2023a)

LLM Ensemble e.g. Gundabathula and Kolar (2024), Barabucci et al.

Application (§4.2) (2024), Coste et al. (2024), Ahmed et al. (2024)

Input Compres- e.g. LLML INGUA (Jiang et al., 2023b), Li

sion (§5.1.1) et al. (2024b), Liu et al. (2023), (Gao, 2024b)
Efficient Com-
putation (§5.1)
Speculative e.g. Stern et al. (2018), Leviathan et al.
Decoding (§5.1.2) (2023), Ou et al. (2024), Huang et al. (2024a)

Mitigating
e.g. Li et al. (2023b), Liu et al. (2021),
Incorrect Knowl-
O’Brien and Lewis (2023), Shi et al. (2024)
edge (§5.2.1)

Knowledge Strengthening
e.g. Tu et al. (2023), Lu et al.
Transfer (§5.2) Correct Knowl-
(2024a), Deng and Raffel (2023)
edge (§5.2.2)

Cooperation (§5)
Supplying New e.g. Ormazabal et al. (2023), (Liu et al., 2024a),
Knowledge (§5.2.3) Zhao et al. (2024b), Zhou et al. (2024b)

e.g. Pan et al. (2023), Huo et al. (2023),

Detector (§5.3.1)
Chen et al. (2023b), Wang et al. (2024c)
Compensatory
Cooperation (§5.3)
Retriever (§5.3.2) e.g. Ma et al. (2023b), Mao et al.
(2024), Li et al. (2024d), Su et al. (2024)

Federated Train- e.g. Fan et al. (2024), Ye et al.

ing (§5.4.1) (2024), Wang et al. (2024d)
Federated Co-
operation (§5.4)
Federated
e.g Zhang et al. (2024b), Li
Prompt Engi-
et al. (2024a), Guo et al. (2022)
neering (§5.4.2)

Figure 3: The primary categorization of LLM collaboration in this survey.

4
their full potential, significantly enhancing their 3 Merging
overall performance and versatility.
Furthermore, LLMs inevitably suffer from com- Single models have inherent limitations, such as po-
putational inefficiencies (Zhou et al., 2024c), hallu- tentially missing important information (Sagi and
cinations (Rawte et al., 2023; Ji et al., 2023; Huang Rokach, 2018), and being prone to getting stuck
et al., 2023), and privacy leaks Fan et al. (2024). in local optima or lacking multi-task capabilities.
Recent studies explore the collaboration strategies To address these limitations, researchers have ex-
between LLMs, which provides potential solutions plored model merging methods, which combine
to mitigate these issues and compensate for their multiple models in the parameter space to create a
shortcomings. unified, stronger model. Model merging has made
significant progress in recent years, with various
The Category of LLM Collaboration Methods techniques cataloged in existing surveys (Li et al.,
Collaboration between LLMs refers to the pro- 2023a). In the era of LLMs, model merging has
cess where multiple LLMs work together, lever- become an important solution for model collabo-
aging their individual strengths and capabilities ration, usually employing basic merging methods
to achieve a shared objective. In this survey, we and demonstrate the effectiveness. This section
categorize LLM collaboration methods into three focuses on the merging techniques that are proven
aspects: merging, ensemble and cooperation. As to be effective for LLMs2 .
shown in Figure 2, Current studies on model merging typically fo-
cus on two key issues: merging to approach a rel-
• Merging involves integrating multiple LLMs
atively optimal solution (M-ROS) and merging to
into a unified, stronger one, primarily through
enhance multi-task capability (M-MTC). Research
arithmetic operations in the model parameter
on M-ROS is based on the finding that gradient-
space.
optimized solutions often converge near the bound-
• Ensemble combines the outputs of different ary of a wide flat region rather than at the central
models to obtain coherent results. Recent stud- point (Izmailov et al., 2018). Model merging of-
ies have proposed various ensemble methods fers a way to approach this relatively optimal point,
tailored for LLMs. thereby yielding a stronger model. M-MTC, on
the other hand, aims to utilize model merging tech-
• Cooperation is a relatively broad concept. niques to enrich a single model with capabilities
This survey focuses on cooperation methods across multiple tasks (Ilharco et al., 2023; Yadav
that leverage the diverse capabilities of differ- et al., 2023). In the following subsection, we will
ent LLMs to accomplish specific objectives, introduce the techniques for each objective and
such as efficient computation or knowledge their application to LLMs.
transfer. It is important to note that for both M-ROS and
It should be noted that as we move from merging M-MTC, current model merging methods are appli-
to ensemble to cooperation, the requirements for cable only to models with the same architecture and
LLMs gradually relax, making the proposed meth- parameters within the same space. Therefore, most
ods increasingly flexible. Specifically, merging candidate models for merging should be trained
methods are effective only when the LLMs share with identical initialization. For instance, the can-
a compatible parameter space, allowing seamless didate models M = {M1 , M2 , · · · , Mk } should
integration. Ensemble methods require LLMs to be fine-tuned from the same pre-trained model M0 .
have diverse yet comparable abilities; without this This requirement ensures compatibility and coher-
balance, the ensemble may be less effective. In ence among the model parameters, promoting suc-
contrast, cooperation methods are more flexible, cessful merging. Unfortunately, for models with in-
focusing on leveraging LLMs with various capabili- compatible parameters, such as LLaMA and QWen,
ties that are specially designed to achieve particular current merging techniques are ineffective.
objectives. 2
Some advanced methods, such as merging after neuron
For each category, we further classify specific alignments - like OT Fusion (Singh and Jaggi, 2020), Re-
methods based on their focus or stages of imple- Basin techniques (Peña et al., 2023; Ainsworth et al., 2023),
and REPAIR (Jordan et al., 2023) - have not been widely
mentation. The comprehensive categorization is explored for LLMs. We leave the implementation of these
shown in Figure 3. techniques on LLMs for future work.

5
3.1 Merging for Relatively Optimal Solution performance on a validation set. Similarly, DiWA
(M-ROS) (Rame et al., 2022) ranks candidate models by their
Machine learning models, particularly deep learn- performance on the validation set and adds new
ing models, often fail to achieve precisely optimal models only if they enhance performance. Typi-
solutions during training (Li et al., 2023a). Re- cally, Greedy Soup and DiWA average the parame-
searchers have demonstrated that the local optima ters of selected models for inference.
of modern deep neural networks are connected by Weighted Average Weighted averaging allows
simple curves, and the paths along these curves can for the assignment of different coefficients to indi-
maintain nearly constant training and test accuracy. vidual models based on their significance or qual-
This indicates that different local optima in the ity, ensuring a better merging. For trained net-
weight space are not isolated but can be connected works with significant weight differences, the sim-
through low-loss paths (Garipov et al., 2018). The ple averaging method often performs poorly (Singh
model obtained by the weighted averaging method and Jaggi, 2020). Therefore, selecting appropriate
can be considered a point on the low-loss path. merging coefficients for different models becomes
Parameter averaging integrates the advantages of a crucial factor. Weighted averaging can be formal-
different models by averaging various local optima, ized as:
reducing the bias and variance of individual models
(Rame et al., 2022). k
X
To improve model performance, M-ROS meth- θ∗ = αi · θMi (5)
ods have been proposed. These methods aim to i=1

combine the parameters of multiple models, merg- where αi refers to the normalized coefficient for
ing relatively optimized points into a better one. We the candidate model Mi .
categorize these methods into two groups: Simple Currently, several methods are available to find
Average and Weighted Average. Although initially the optimal merging coefficients. For instance,
developed for small deep models, these techniques Learned Soup (Wortsman et al., 2022) optimizes
are also effective for LLMs. We will first introduce the mixing coefficients on a validation set to min-
the basic M-ROS methodologies and then discuss imize the loss function, thereby combining the
their application to LLMs. strengths of multiple models to enhance overall
3.1.1 Basic M-ROS Methodologies performance. Matena and Raffel (2022) propose
to utilize Fisher Information Matrix to measure
Simple Average Simple parameter averaging is
the importance of parameters of models fine-tuned
a kinds of methods used to combine the param-
with different random seeds, and employ the impor-
eters of multiple fine-tuned models with equal
tance scores as coefficients to merge them. Further-
coefficients (Guo et al., 2023), thus creating a
more, Jang et al. (2024) propose a method based
stronger one (Singh and Jaggi, 2020). Empirical
on geometric relationships, interpolating fine-tuned
evidence shows that these fused models often out-
models based on the angular divergence between
perform individual models in terms of accuracy,
parameters.
robustness and stability. For k candidate models,
M = {M1 , M2 , · · · , Mk }, simple parameter av- 3.1.2 Adaptation to LLMs
eraging can be formalized as: The above model merging techniques have been
k successfully adapted to LLMs for the objective of
∗ 1X
θ = θMi (4) acquiring stronger LLMs and enhancing RLHF.
k
i=1
Acquiring Stronger LLMs To obtain stronger
where θMi refers to the parameters of i-th model LLMs, some studies propose novel methods tai-
and θ ∗ is the merged parameters. lored for LLMs. Wan et al. (2024b) suggest calcu-
To maximize the benefits of multiple models, lating the merging coefficients based on the vari-
Model Soup (Wortsman et al., 2022) introduces ation ratio of parameter matrices before and af-
Uniform Soup and Greedy Soup. Uniform Soup ter fine-tuning. Similarly, Liu et al. (2024b) pro-
simply averages the model parameters. Greedy pose leveraging LLM checkpoints saved during
Soup adds models to the pool one at a time, ensur- pre-training, in conjunction with Bayesian opti-
ing each new model either improves or maintains mization, to navigate the extensive search space

6
 Model 1 Model 2 eter spaces. Such divergence often reflects task-
Parameter
related information. Consequently, M-MTC meth-
Task 𝒕𝟏 Vector - 𝝉𝒕𝟏
ods aim to relieve the divergence and achieve a
Task 𝒕𝟐 Vector - 𝝉𝒕𝟐 balanced merging of models with different capabil-
ities, thereby producing a single model capable of
Conflicts
handling multiple tasks.
Figure 4: Illustration of the parameter conflicts. The di- Early studies addressed the issue of divergence
rection denoting sign and length denoting magnitude of by using different merging coefficients for various
task vectors. The conflicts occur when the task vectors
models (§3.2.1, Weight Average), while current re-
have opposite signs.
search prefers to extract task properties from the di-
vergence to achieve more flexible merging (§3.2.2,
and identify optimal merging coefficients. To ad- Task Property). Furthermore, recent works have
dress concerns about gradient mismatch, Lin et al. started to employ incremental learning techniques
(2023) introduce an adaptive method that assigns to enhance model merging performance (§3.2.3, In-
different combination ratios to various layers of cremental Training). We separately introduce these
the model, optimizing these combinations to bal- methods in the following subsections.
ance human preference alignment and pre-training
3.2.1 Methods based on Weighted Average
proficiency.
Some studies adopt weighted-average strategies (as
Other studies employ existing model merging
introduced in §3.1.1) to adjust the importance of
techniques to create stronger LLMs for specific ob-
different models. Jin et al. (2022) propose Reg-
jectives. For example, Fu et al. (2024a) propose a
Mean to selectively integrate the linear layers of
disperse-then-merge framework, which first train
Transformer models while using simple averag-
multiple sub-models using different instruction-
ing for other layers, thereby minimizing the di-
tuning data portions and then fuse them into a sin-
vergence between the merged model and multiple
gle LLM with weighted merging.
models fine-tuned on various datasets. Daheim
Enhancing RLHF Model merging techniques et al. (2023) advocate refining model merging using
also help improve the alignment of LLMs with hu- estimates derived from the Hessian matrix, facilitat-
man preferences. Lin et al. (2023) present an adap- ing more precise adjustments to model parameters.
tive method where different combination ratios are Nathan et al. (2024) combine Fisher weighted av-
assigned to various layers of the model, optimizing eraging with model pruning, achieving efficient
these combinations to balance human preference model merging.
alignment and pretraining proficiency. Ram’e et al.
3.2.2 Methods based on Task Property
(2024) propose fine-tuning multiple reward mod-
els and then averaging their parameters to create a Merging methods based on weighted average em-
superior reward model that aligns better with hu- phasize the importance of parameters but overlook
man preferences. Similarly, Fu et al. (2024b) use their task-specific properties, leading to significant
a weighted averaging approach to improve align- performance degradation in certain tasks. Ilharco
ment in LLMs during the supervised fine-tuning et al. (2023) find that “Simple Averaging suffers
(SFT), effectively reducing the impact of data bias. from a 10% performance drop”. To address this
Lu et al. (2024b) propose to use model merging issue, recent studies introduce a new paradigm
technique to reduce alignment taxes during RLHF known as the task vector. Ilharco et al. (2023)
training for LLMs. define the task vector τt as “a vector specifies a
direction in the parameter space of a pre-trained
3.2 Merging for Enhancing Multi-Task model, such that movement in that direction im-
Capability (M-MTC) proves performance on the task”, which can be
formalized as:
Recently, some studies have attempted to merge
models with different capability to construct a uni- τt = θtft − θ pre (6)
fied model with multi-task capability (Li et al.,
2022). Typically, these models are fine-tuned from where θtft refers to the parameters fine-tuned with
the same pre-trained model but with different task- the specific data for task t, and θ pre refers to the
specific data, leading to divergence in their param- original parameters of the pre-trained model.

7
 Task vector can more effectively resolve param- niques to reduce these conflicts before merging.
eter conflicts during the model merging. As illus- Such methods further enhances the relevance of pa-
trated in Figure 4, using the pre-trained model as rameters to the task, and we introduce these meth-
a reference, the variation in the direction of task ods separately. DARE (Yu et al., 2023), a tech-
vectors of the fine-tuned models indicates the pres- nique that efficiently reduces redundancy in fine-
ence of conflicts in the parameter space. To address tuned language models by dropping and rescaling
parameter conflicts problem, recent studies aim to parameters. DELLA- MERGING (Deep et al., 2024)
exploring methods to mitigate conflicts and strike further selects important parameters for fusion on
a balance between the different models. Parame- the basis of DARE. As domain-specific data and
ter Conflict methods resolve parameter conflicts at training techniques grow, the distinctions between
same position of parameters, while Fewer Parame- fine-tuned models and their base models become
ter methods identify and prune redundant parame- more significant. However, DARE experiences sig-
ters to reduce conflict. In addition, we introduce a nificant performance drops, resulting in insufficient
tool that includes some methods in Toolkit. capability to process multiple domains effectively.
DPPA (Zhu et al., 2024) presents a dual-stage prun-
Resolving Parameter Conflicts TASK A RITH -
ing approach as Dynamic Pruning Partition Ampli-
METIC (Ilharco et al., 2023) initially derives task
fication (DPPA) for effectively merging divergent
vectors through arithmetic operations between fine-
fine-tuned large language models across different
tuned and pre-trained parameters, as shown in equa-
domains.
tion (6). These task vectors are subsequently used
to merge models for enhancing performance on Toolkit Recently, Goddard et al. (2024) have
target tasks. To further address the issue of param- developed Arcee’s MergeKit, an open-source
eter conflicts in model merging, T IES -M ERGING toolkit that integrates various model merging meth-
(Yadav et al., 2023) identifies two primary causes ods, including Model Soups, DARE, and TIES-
of interference: redundant parameter values and M ERGING. This toolkit significantly advances the
sign inconsistencies across models. Building on application of model merging strategies in LLMs3 .
these advancements, A DA M ERGING(Yang et al.,
2023) reduces conflicts even further by consid- 3.2.3 Methods based on Incremental Training
ering the importance of different model parame- The aforementioned methods still suffer from per-
ters. M ETAGPT (Zhou et al., 2024a) proposes effi- formance degradation. Therefore, several ap-
cient methods based on A DA M ERGING and TASK proaches involving incremental training have been
A RITHMETIC. Akiba et al. (2024) propose a merg- proposed to restore their original performance.
ing method for LLMs, which not only employ T IE - Concrete TA/AM (Tang et al., 2023) aims to find
M ERGING for merging in parameter space, but also a shared low-dimensional subspace within the
adopt evolutionary algorithms to optimize the data model parameter space to minimize task interfer-
inference path inside the merge model. The above ence without significantly impacting performance.
methods have been successfully adapted to LLMs. Surgery (Yang et al., 2024a) introduces a represen-
Kim et al. (2024) apply the above methods to fuse tation surgery technique to mitigate representation
the LLMs obtain a stronger LLM evaluator. Ham- bias in multi-task model fusion.
moud et al. (2024) investigate the effects of above
methods on LLM safety alignment. 4 Ensemble
Unlike the aforementioned task vector based Ensemble learning is another effective collabora-
methods to resolve conflicting parameters, Stoica tion strategy that differs from model merging meth-
et al. (2024) propose Z IP I T that retains similar pa- ods by focusing on the combination of model out-
rameters from another perspective. Z IP I T first iden- puts. Traditional techniques like Adaboost (Freund
tifies highly correlated parameters between differ- and Schapire, 1997), Bagging (Breiman, 1996),
ent models. It then merges these parameters while and Stacking (Wolpert, 1992) have significantly
retaining significantly different layers, thus improv- advanced machine learning research. In the era
ing the merging flexibility. of LLMs, ensemble learning continues to be cru-
Pruning Redundant Parameters Given that cial, enhancing the overall performance of various
conflicts may exist in the parameters of different LLMs.
3
models, another solution is to employ pruning tech- [Link]

8
 However, LLMs typically solve various tasks Shnitzer et al. (2023) take the lead to explore
through text generation, resulting in more flexi- the feasibility and limitations of learning routers by
ble and naturally expressive outputs. Therefore, using various benchmark datasets. Lu et al. (2023)
traditional ensemble methods designed for classi- introduce Z OOTER, a system that first employs a
fication tasks cannot be directly applied to LLMs. reward model to calculate scores for query-output
To address this issue, many studies explore spe- pairs using the training set. These scores are then
cific ensemble methodologies tailored for various utilized to train a router using the knowledge dis-
LLMs4 . Additionally, the benefits of ensemble tillation strategy, allowing it to select the optimal
learning have inspired research into various appli- LLM based solely on input queries. Ding et al.
cations of these techniques. In the following sec- (2024) employ a router that assigns queries to ei-
tions, we will separately introduce LLM ensemble ther a small model or LLMs based on the predicted
methodologies and applications in detail. query difficulty and the required quality level, sig-
nificantly reducing inference costs. Srivatsa et al.
4.1 LLM Ensemble Methodology (2024) investigate the feasibility of classifier-based
For different inputs, LLM that performs best is not and clustering-based routing methods for LLMs.
always the same, prompting extensive research into Inspired by self-play in reinforcement learning,
ensemble methods for LLMs. Unlike classification- Mohammadshahi et al. (2024) train the router by re-
based machine learning models, LLMs typically cycling the self-produced triplets, (query, response,
generate a sequence of tokens as output. This out- score). Unlike previous studies, Lu et al. (2024c)
put is often discrete, making direct ensemble chal- integrate multiple Chat LLMs by randomly select-
lenging. Additionally, structural differences be- ing an LLM at each turn in the dialogue, rather than
tween various LLMs result in vocabularies and learning a router. To effectively evaluate the router
output distributions that are difficult to unify, fur- capability and limitations, Hu et al. (2024) pro-
ther complicating ensemble strategies (Xu et al., pose a new benchmark, ROUTER B ENCH, mainly
2024b). focusing on performance and economic cost.
Since ensemble generally occurs during the infer-
ence period, we categorize and introduce existing 4.1.2 Ensemble D URING Inference
ensemble methods employed B EFORE, D URING, During the inference period, LLMs generate to-
and A FTER the inference period. As illustrated kens auto-regressively. This process often results in
in Figure 5, ensemble methods B EFORE inference early errors compounding over time, causing subse-
select the most suitable LLM for different input quent tokens to deviate from the intended meaning
examples, ensemble methods D URING inference (Ranzato et al., 2016) and leading to hallucinations
combine outputs at each decoding step, and ensem- (Zhang et al., 2023a).
ble methods A FTER inference aim to select the best To address this problem, some studies perform
response from several outputs generated by various ensemble LLMs at each decoding step. Li et al.
LLMs. (2024c) combine untrusted LLMs with a benign
smaller LLM by weighted-averaging the output
4.1.1 Ensemble B EFORE Inference distributions, mitigating issues such as copyright
Such methods aim to select the best LLM for spe- infringement, data poisoning, and privacy viola-
cific examples before inference. Similar but dif- tions. Hoang et al. (2023) interpolate the output
ferent from various Mixture-of-Expert (MOE) ap- distributions from the machine translation model
proaches (Jacobs et al., 1991; Collobert et al., 2003; and the LLM, boosting translation performance.
Eigen et al., 2013; Fedus et al., 2022; Jiang et al., Wang et al. (2024b) formulate the Frugal Fusion of
2024), which learn sparse networks from scratch, Experts problem and proposes an efficient fusion
ensemble B EFORE Inference focuses on training method by addressing it as a graph shortest path
external routers (Rosenbaum et al., 2017) for sev- problem. These methods require the ensemble to
eral pre-trained LLMs to achieve optimal LLM occur among LLMs which must at least have the
selection. same vocabulary. This restriction ensures that the
4
It is worth noting that many studies explore self-ensemble output distributions are aligned and can be interpo-
approaches, such as self-consistency (Wang et al., 2023b) lated effectively.
and rationale augmentation (Wang et al., 2022). Our survey
focuses on ensembles across different LLMs and does not However, most open-source LLMs are hetero-
cover these methods. geneous and have different vocabularies, hinder-

9
 (a) Ensemble before Inference (b) Ensemble during Inference (c) Ensemble after Inference

Isaac Newton Issac Isaac Newton

Selection

Issac New Issac Isaac Newton Isaac Newton

Router Newton
Network Decoding Step - t

Who discovered the law of gravity? Who discovered the law of gravity? Who discovered the law of gravity?

Figure 5: The illustrations of LLM ensemble methods B EFORE (a), D URING (b), A FTER (c) inference.

ing direct ensembling. To address this issue, Fu LLMs. Lee et al. (2023) select the best instruc-
et al. (2023) employ dynamic programming to re- tion from several candidates produced by LLMs
cursively minimize the total cost measured by the for instruction-tuning data construction. Jiang et al.
exact match scores of editing one sequence of to- (2023a) explore various unsupervised metrics for
kens to match another. To further enhance the suc- selection, including BERTScore (Zhang* et al.,
cess rate of token alignments, Wan et al. (2024a) 2020), BLEURT (Sellam et al., 2020), BARTScore
replace the exact match constraint with a minimum (Yuan et al., 2021), and ChatGPT scores. However,
edit distance strategy. Mavromatis et al. (2024) they find that the effectiveness of the selection is
follow the above token alignments, proposing the constrained by the quality of candidate pools. To
use of perplexity to compute the coefficients for the address this problem, Jiang et al. (2023a) adopt an
outputs of diverse LLMs during ensemble. Xu et al. additional fusion model to generate the final output
(2024b), Huang et al. (2024c), and (Yu et al., 2024) using the top-ranked candidates as inputs.
consider overlapping tokens as anchors to project
the output distribution produced by heterogeneous 4.1.4 Discussion about LLM Ensemble
LLMs into the same space. Specifically, Xu et al. Methods
(2024b) propose to directly learn the projection ma- The methods described above focus on ensemble
trices between different vocabularies using the an- techniques at different stages of inference, each
chors as bridges, while Huang et al. (2024c) and Yu with its own strengths and weaknesses. We dis-
et al. (2024) calculate the relative representations cuss them in terms of inference speed, ensemble
from anchors to different vocabularies, thereby in- granularity, and limitations.
directly achieving the vocabulary projection.
Inference Speed Almost all ensemble methods
4.1.3 Ensemble A FTER Inference decrease inference speed. Ensemble methods B E -
FORE inference slightly slow down the process, as
The final LLM ensemble methods combine the gen-
they involve selecting the optimal LLM via addi-
erated outputs A FTER the inference period.
tional routers. Ensemble methods D URING infer-
One approach to achieving the LLM ensemble ence require each LLM to perform forward compu-
involves building LLM cascades to reduce the in- tation for test examples. If we have k LLMs, the
ference cost associated with using large LLMs ex- inference speed will slow down by a factor of k,
clusively. Chen et al. (2023d) employ a sequence although this can be mitigated by using k times the
of LLMs ranked by the number of parameters to number of GPUs. Ensemble methods A FTER in-
generate outputs, halting the process and returning ference not only require k times the computational
results once a preceding smaller LLM produces cost but also spend additional time on output selec-
outputs of sufficient quality. Yue et al. (2024) pro- tion or fusion (Jiang et al., 2023a), resulting in the
pose verifying the correctness of answers generated lowest inference speed.
by a smaller LLM first and utilizing LLMs to solve
the problem if the initial answer is incorrect. Ensemble Granularity The aforementioned en-
Another line of research focuses on selecting the semble methods operate at different levels of gran-
best candidate from several produced by various ularity. Ensemble methods B EFORE and A FTER in-

10
 Inference Stage Inference Speed Ensemble Granularity Limitations & Challenge
B EFORE ↘ example-level Accuracy of Routers
D URING ↘↘ token-level Heterogeneous Architectures
A FTER ↘↘↘ example-level Accuracy of Candidate Pools

Table 1: The characteristic of LLM ensemble methods employed at different inference stages.

ference typically work at the example level, provid- Gundabathula and Kolar (2024) adopt LLM en-
ing a coarse-grained ensemble. Ensemble methods semble to enhance the clinical text error detection
D URING inference perform ensemble at the token and correction. Oniani et al. (2023) and Barabucci
level, offering a fine-grained approach. Since pre- et al. (2024) utilize majority voting and average
ceding tokens often influence succeeding tokens, weighting, respectively, to select the most frequent
this fine-grained ensemble can reduce exposure candidate disease.
bias (Ranzato et al., 2015; Xu et al., 2020) and de-
LLM Ensemble for Overestimation Mitigation
crease hallucination in LLMs, thus holding better
in RLHF Ensemble learning can alleviate the
potential for performance enhancement.
poor calibration and unreliable prediction problems
Limitations Each category of ensemble meth- of LLMs. Therefore, Eisenstein et al. (2023), Coste
ods have specific limitations, as shown in Table et al. (2024) and Ramé et al. (2024) perform en-
1. Although ensemble methods B EFORE inference semble with multiple reward LLMs to mitigate the
offer better speed, they require additional training overoptimization problem in RLHF. Considering
of the router. The data used for router training that fine-tuning reward models based on LLMs can
can limit the generalization and performance of be computationally expensive, Zhang et al. (2024c)
these ensemble methods (Shnitzer et al., 2023). En- and Zhai et al. (2023) respectively employ light-
semble methods D URING inference are typically weight LoRA (Hu et al., 2022) to adapt the LLM to
constrained by the heterogeneous architecture of multiple reward models. Ahmed et al. (2024) pro-
LLMs. For example, differences in vocabularies pose using a shared LLM but separate linear heads
can lead to non-corresponding output distributions, for reward ensemble. These methods successfully
hindering direct ensemble (Huang et al., 2024c; Yu reduce the overestimation in RLHF and improve
et al., 2024). Ensemble methods A FTER inference the alignment performance..
often require multiple LLMs to generate several
5 Cooperation
candidates, construct a candidate pool, and then
select or reorganize the final output. Thus, the ac- In the era of LLMs, collaborative strategies ex-
curacy of the candidate pool (Jiang et al., 2023a) tend beyond mere merging or ensemble. Increas-
and the selection strategy are the main limitations. ingly, studies are focusing on broader approaches to
solving various problems or specific tasks through
4.2 LLM Ensemble Application cooperation between LLMs. In the following sec-
In addition to methodological research, many stud- tions, we introduce the cooperation strategies based
ies apply LLM ensembles to specific applications. on their objectives: efficient computation (§5.1),
This is due to the capability of ensemble learning knowledge transferring (§5.2), compensatory co-
ability on specific tasks, domains and strong cali- operation (§5.3), federated cooperation (§5.4).
bration. We categorize these related studies based
5.1 Efficient Computation
on their objectives.
As LLMs grow in scale, the computational re-
LLM Ensemble for Specific Tasks or Domains sources required for their inference increase signifi-
Ensemble learning for LLMs are typically em- cantly. Consequently, accelerating model inference
ployed for specific tasks. Si et al. (2023) improve has become an urgent necessity. Smaller LLMs
LLM reasoning performance by training a classi- play a crucial role in accelerating larger LLMs due
fier to select the best answer generated by various to their lightweight nature (Miao et al., 2023). Re-
reasoning experts. Gundabathula and Kolar (2024) search on model acceleration through cooperation
employ LLM ensemble for SQL generation. Some can be divided into two main categories: Input
studies employ LLM ensemble for medical tasks. Compression (§5.1.1) and Speculative Decoding

11
(§5.1.2). Input Compression achieves efficient com- Among the above-mentioned methods, they usually
putation by using smaller LLMs to compress in- need to cooperate with a compression model, such
puts, thereby reducing the context length. Specula- as a summarization model, to achieve the input
tive Decoding involves leveraging smaller LLMs compression.
to draft multiple tokens speculatively, with larger
LLMs verifying these draft tokens in parallel. Prompt Pruning The core idea behind the
prompt pruning is to remove unimportant tokens,
5.1.1 Input Compression sentences, or documents, where the cooperation
model targets at prompt pruning. Ali et al. (2024)
utilize graph-construct model to obtain a graph
from the textual information in the prompt and ex-
tracting key information elements from the graph
Input Compression
to obtain the compressed prompt. Pan et al. (2024)
introduce the LLMLingua-2 distillation method,
which classifies each input token using a Trans-
former encoder. Then it preserves the top N to-
LLM
kens with the highest classifying probability, cap-
turing all key information for prompt compres-
sion. (Huang et al., 2024b) propose a coarse-to-fine
pruner that initially identifies crucial CoT exam-
Figure 6: LLMs cooperate with compression module
for input compression. ples from a large batch and then further prunes the
unimportant tokens.
Input compression aims to use small models to Prompt Summarization The core idea of
compress input prompts or long text content. When prompt summarization is to condense the original
processing long input sequences, the self-attention prompt into a shorter summary which preserves the
mechanism of LLMs results in significant increases same semantic information, where the cooperation
in computational cost and memory usage due to model is an extractive or abstractive summarization
its quadratic time complexity (Xu et al., 2024a). model. Liu et al. (2023) propose two compression
Input compression reduces computational costs, methods. The first method trains a summarization
memory usage, and access costs by shortening in- model to compress the context. The second method
put lengths and reducing the number of tokens the further reduces the number of tokens by deleting
model needs to handle during the pre-fill phase, words with less semantic impact. RECOMP (Xu
thereby improving memory efficiency (Wan et al., et al., 2023) introduces an abstractive compressor
2024c). Additionally, input compression acceler- that takes a question and retrieved documents as in-
ates token generation in autoregressive decoding by put to produce a concise summary. Then the LLM
shortening input sequence lengths, thus enhancing generate the answer based on the concise summary.
inference efficiency. SemanticCompression (Fei et al., 2023) proposes a
Zhou et al. (2024c) propose a review on effi- method that breaks text into sentences, groups sen-
cient inference, where input compression being a tences by topic, and then summarizes each group
significant part of the review. Input compression with pre-trained model. Finally, the LLM generate
mainly focuses on prompt pruning, prompt sum- the response based on all the groups. To address the
marization, and soft prompt compression. The different tasks, Li et al. (2024b) propose PCToolkit,
core idea of prompt pruning is to delete unim- which consists compressors that are developed with
portant tokens, sentences, or documents in the in- different targets.
put prompt based on predefined or learnable im-
portance metrics. Prompt summarization aims to Soft Prompt Compression Soft prompt com-
compress the original prompt into a shorter sum- pression involves using virtual tokens to assist in
mary while retaining similar semantic information. prompt compression, where the cooperation model
Soft prompt compression involves designing a soft is an text feature extractor. For example, SelfCP
prompt which is a sequence of learnable contin- (Gao, 2024a) uses a frozen LLM as an encoder and
uous tokens. It is much shorter than the original decoder, inserting special tokens into the prompt to
prompt and will be utilized as the input for LLMs. generate virtual tokens, thereby achieving prompt

12
compression and response generation collaboration. model predictions, some studies have introduced
(Gao, 2024b) projects candidate demonstrations early stopping mechanisms. This method termi-
into virtual tokens via a LLM during in-context nates the prediction process early based on specific
learning, then it selects appropriate demonstrations conditions, thereby saving computational resources.
based on semantic similarity. Finally, it generates For example, Huang et al. (2024a) and Liu et al.
the response using a frozen LLM. (2024c) propose using probabilistic predictions to
In general, we can observe that from prompt assess the acceptance rate of draft model’s hypothe-
pruning to prompt summarization to soft prompt ses, deciding whether to continue generating more
compression, the compression ratios increase drafts based on predefined thresholds. Furthermore,
which results in higher efficiency. However, the Liu et al. (2024d) incorporate an early stopping
corresponding information loss is also greater. mechanism after the initial N layers of the model
during draft hypothesis generation.
5.1.2 Speculative Decoding

5.2 Knowledge Transfer via Cooperation

LLMs typically encompass various capabilities.

However, due to the difficulty in obtaining training
Draft Generator data and the high training costs, directly transfer-
ring knowledge or capabilities from one LLM to
another has attracted significant attention. Consid-
ering that the output probabilities of LLMs often
LLM contain the embedded knowledge of the models,
recent methods have primarily focused on trans-
ferring knowledge between LLMs. For example,
Wan et al. (2024a,b) transfer knowledge from mul-
Figure 7: LLMs cooperate with draft generator for spec- tiple LLMs to a target model via continued training
ulative decoding. with knowledge distillation (Hinton et al., 2015).
Nonetheless, most of the recently proposed meth-
The primary focus of decoding acceleration is to ods focus on cooperation at the inference stage
cooperate smaller models to generate drafts, which without involving training and can be categorized
are then verified by larger models to expedite infer- based on their objectives: Mitigating Incorrect
ence (Stern et al., 2018). This approach is currently Knowledge, Strengthening Correct Knowledge, and
referred to as speculative decoding (Leviathan et al., Supplying New Knowledge.
2023; Xia et al., 2023). Xia et al. (2024) summa-
rize various drafting strategies, including Indepen- 5.2.1 Mitigating Incorrect Knowledge
dent Drafting and Self-Drafting. The drafted to-
kens are then verified in parallel by the target LLM, Hallucination (Rawte et al., 2023; Huang et al.,
with verification methods such as Greedy Decoding 2023) and bias (Wang et al., 2024a) are common is-
(Gu et al., 2017) and Nucleus Sampling (Holtzman sues in LLMs. Ji et al. (2023) argue that LLMs
et al., 2020b). The acceleration effect of specula- trained with the next-word prediction objective
tive decoding largely depends on the acceptance store superficial patterns instead of recognizing
rate of drafted tokens at each step. This acceptance real-world knowledge extracted from the training
rate is influenced by several factors, including the corpora. Therefore, many studies aim to explore
quality of the drafts, the verification criteria, and cooperation methods for removing incorrect knowl-
the behavioral consistency between the drafter and edge existed in the output logits.
the target LLM. Drawing on the observation that the shortcom-
For instance, Ou et al. (2024) propose the use ings of larger LLMs are even more pronounced in
of an adaptive N-gram model based on the current smaller ones, Li et al. (2023b) propose Contrastive
context for the rapid drafting phase, followed by a Decoding (C D). C D encourages token yi selection
verification phase where the original LLM evalu- from the delta distribution between LLMs and their
ates and confirms the proposed tokens. In addition corresponding weaker, amateur models, thereby
to using draft models to reduce the number of actual eliminating incorrect knowledge from output distri-

13
 Who discovered the law of gravity? 5.2.2 Strengthening Correct Knowledge
Beside mitigating hallucination or bias from the
Hmm, his answer output distributions, another line of research fo-
doesn't seem correct.
I need to reconsider it.
cuses on enhancing the faithfulness of decoding
outputs to the input or instructions through LLMs
Albert! cooperation. This typically involves using addi-
tional models to strengthen the correct knowledge
- increasing the likelihood of potentially correct
outputs. This line of research can be traced back
Amateur Model Target LLM
to studies on attribute-controlled text generation,
which encourage language models to output tokens
Figure 8: Illustration of core idea of C D. The amateur that effectively predict the input attributes, thereby
model is more prone to errors, thus the target LLMs
improving [Link] examples include
needs to reconsider the answer accordingly.
F UDGE (Yang and Klein, 2021) and G E D I (Krause
et al., 2021), which are based on Bayesian factor-
butions: ization:
yi ∼ log pLLM (yi |y<i ) − log pAMA (yi |y<i ) (7)
| {z } | {z } p(yi |y<i , c) ∝ p(yi |y<i ) · p(c|yi , y<i ) (8)
vanilla distribution amateur distribution
yi ∼ log pLLM (yi |y<i ) + log pV ER (c|yi , y<i ) (9)
This work has significantly inspired subsequent | {z } | {z }
vanilla distribution verification
studies. Some studies employ C D to enhance rea-
soning capabilities. O’Brien and Lewis (2023) where c is the input attribute and log p(c|yi , y<i ) is
demonstrate that C D improves reasoning for LLMs derived from additionally learned attribute predic-
by preventing certain abstract reasoning errors. tors, namely verifier. It should be noted that any
Phan et al. (2024) utilize distillation techniques suitable model, including the LLM itself, can serve
to obtain amateur models and perform contrastive as the verifier.
decoding to enhance reasoning capabilities. Ad-
ditionally, some studies use C D for trustworthy
Who discovered the law of gravity?
generation. Waldendorf et al. (2024) experiment
with C D in various amateur models for machine I am confused.
translation, showing that C D reduces hallucinations “Albert” and “Isaac” seems
both correct.
in large multilingual machine translation models.
Liu et al. (2021) employ C D for language detoxifi- I prefer “Isaac”.
cation and sentiment-controlled generation. Zhang
et al. (2023b) and Niu et al. (2024) induce halluci-
nations or toxic content into LLMs by fine-tuning Target LLM LLM Verifier
with non-factual or toxic examples and then use
these models as amateurs for contrastive decoding. Figure 9: Illustration of core idea of verification based
Qu et al. (2024) leverage counterfactual C D for methods. The LLM verifier needs to check and assist
distractor generation. Some studies explore diverse selecting the correct output from candidates.
contrastive strategies. Shi et al. (2024) observe
that output distributions from an MoE model us- Recently, Deng and Raffel (2023) employ an
ing different routing strategies differ substantially, auxiliary reward model as the verifier to encour-
thus they utilize unchosen experts as the amateur age LLMs to generate text that has certain prop-
model to achieve contrastive decoding. Yuan et al. erties. Tu et al. (2023) verify each decoding posi-
(2023) combine C D with speculative decoding to tion according to equation (9) in language genera-
achieve both acceleration in decoding and quality tion tasks, enhancing the faithfulness of LLMs. To
improvements. achieve accurate verification with sufficient infor-
Although C D can help mitigate incorrect knowl- mation, Lu et al. (2024a) introduce D IVER, which
edge during inference, LLMs and the correspond- employs dynamic token spans that can potentially
ing amateur models typically belong to the same be generated to calculate point-wise mutual infor-
family and require aligned output distributions. mation (PMI) for verification. Additionally, they

14
demonstrate that using smaller LLMs for verifica- Who discovered the law of gravity?
tion can alleviate the decrease in inference speed
“Albert”.
without significantly affecting performance.

5.2.3 Supplying New Knowledge You are wrong.

Recently, some studies have observed that the vari-

ation of output logits reflect the variation of LLMs
capabilities. Inspired by this, they propose to tune Target LLM Detector

larger LLMs without training. Instead, they mod-

Figure 10: LLMs cooperate with detector.
ify the output logits to provide new capabilities
extracted smaller LLMs:
for ensuring the reliability and credibility of gener-
log pL ARGE -C (yi |y<i ) ∝ log pL ARGE (yi |y<i )
| {z } ated content. We introduce related work from two
vanilla distribution types of hallucinations: factual hallucinations and
+ [log pS MALL -C (yi |y<i ) − log pS MALL (yi |y<i )] faithfulness hallucinations.
| {z }
capability reflection
(10) Factual Hallucinations. Factuality hallucination
emphasizes the discrepancy between generated con-
where L ARGE -C and S MALL -C refers to models tent and verifiable real-world facts (Bai et al., 2024).
with specific capability while L ARGE and S MALL Current mainstream methods for detecting factual
refers to the original models. hallucinations can be divided into: 1) retrieve exter-
Ormazabal et al. (2023) and Liu et al. (2024a) nal facts: identifying factual inaccuracies by com-
separately propose emulated tuning and proxy tun- paring the model-generated content against exter-
ing, extracting Chat capability from smaller LLMs nal knowledge sources (Min et al., 2023b; Tang
and integrate the capability into larger LLMs with et al., 2024). 2) uncertainty estimation: retrieving
the aforementioned equation (10). Zhao et al. external facts introduces additional cost. Therefore,
(2024b) employ safe and unsafe smaller models several methods rely on internal knowledge to alle-
to adversarially modify a significantly larger safe viate hallucinations (Chen et al., 2024; Tsai et al.,
model’s decoding probabilities, achieving jailbreak- 2024). Among them, the cooperation studies of
ing for LLMs. LLMs mainly falls into the first category.
The above methods require that both small and Some studies directly utilizes additional models
large LLMs share the same vocabulary. To address as detectors to identify fact errors in the output
this issue, Zhou et al. (2024b) introduce weak-to- of LLMs. RARR (Gao et al., 2022) leverages a
strong search, utilizing the log-likelihood differ- NLI model to evaluate the factulity between the
ence between small tuned and untuned LLMs as generated sentences and evidence. Consider that
rewards to guide the decoding of larger LLMs with some evidence may not fully support a claim, thus
tree search, mitigating the need for shared vocabu- Factcheck (Wang et al., 2023c) identifies the stance
laries. of evidences and gathers the relevant evidence,
then it uses RARR to determine whether the gath-
5.3 Compensatory Cooperation ered evidence supports the original text. Addition-
In practical applications, large models still face ally, to detect the factuality of the model regarding
some uncontrollable issues due to their lack of in- complex multi-step problems, PROGRAMFC (Pan
terpretability (Zhao et al., 2024a). Therefore, it et al., 2023) decomposes complex claims into sim-
is necessary to introduce additional controllers to pler sub-tasks that can be solved using question-
compensate for the deficiencies of LLMs. Depend- answer module, NLI module, in-context learning
ing on the desired properties, controllers can be module. However, single-pass verification can-
served as: 1) detector, 2) retriever, etc.. not fully utilize the experience gained from sim-
ilar cases and may repeat past errors. Therefore,
5.3.1 Detector SALAM (Wang and Li, 2023) exploits a auxiliary
LLMs may generate incorrect responses or halluci- model to assist the main LLM in learning from
nations due to a lack of relevant knowledge. There- mistakes through interactive cooperation model.
fore, detecting hallucinations in LLMs is crucial Other studies exploits auxiliary models to col-

15
lect relevant evidence for verification (Chen et al., from QAFactEval (Fabbri et al., 2022).
2023c; Huo et al., 2023; Ousidhoum et al., 2022; In brief, by collaborating with a different detec-
Yao et al., 2022). Huo et al. (2023) adopt infor- tors, LLMs can often verify or correct some errors,
mation retrieval model to obtain related passages thereby alleviating the issue of uncontrolled con-
for factual detecting. The retrieved documents con- tent generation.
tain up to several thousand words, which becomes
5.3.2 Retriever
cumbersome for both humans and models to make
a judgment based on them. Therefore, Chen et al. Who discovered the law of gravity?
(2023c) compress retrieved text with an summa-
rization model and obtains relevant claims, which I don’t know. I need
relevant materials.
is provided for verifying the fact of LLMs output.
The law of gravitation is
Faithfulness Hallucinations. Faithfulness hallu- a basic law in physics.
“Isaac” proposes the
cination refers to the divergence of generated con- law of gravitation.
tent from user instructions or the context provided
by the input (Bai et al., 2024). The current main- Target LLM Retriver
stream methods for detecting faithfulness hallucina-
tions can be divided into: 1) rule-based: which mea- Figure 11: LLMs cooperate with retriever.
sures the overlap of pivotal facts between the gen-
erated content and the source content (Nan et al., The information possessed by LLMs is limited.
2021; Goodrich et al., 2019); 2) classifier-based, To extend the knowledge of large models, retrieval
leveraging trained classifiers to obtain the degree augmentation generation (RAG) (Gao et al., 2024)
of entailment between the generated and the source aims to use external data sources to assist text gen-
content (Falke et al., 2019; Mishra et al., 2021); eration. Collaborative models serve as bridges to
3) QA-driven, employing question-answering sys- external knowledge, assisting in the retrieval of in-
tems to verify the consistency between the gener- formation. The cooperation models for retrieving
ated and the source content; 4) uncertainty esti- different types of data vary significantly, including:
mation, which evaluates faithfulness by assessing 1) unstructured data; 2) structured data.
the model’s confidence in its generated outputs; For unstructured data, such as text, some stud-
5) prompting-based, where LLMs are prompted to ies retrieve world knowledge to expand the corpus
function as evaluators, assessing the faithfulness of the LLMs. Izacard et al. (2022) start by re-
of generated content through specific prompting trieving the top-k relevant documents from a large
strategies. Among these, the collaborative stud- corpus of text with Contriever (Izacard et al., 2021).
ies of LLMs mainly falls into categories 2) and 3). Then, these documents are fed to the LLM, along
Their difference mainly lies in the use of different with the query, which in turns generates the out-
cooperation models for hallucination detection. put. Besides, it is inconvenient to train language
The classifier-based methods employ NLI mod- models with special cross attention mechanisms
els to filter out generated sentences that is not sup- to encode the retrieved text. Thus REPLUG (Ma
ported by the fact of evidence document (Chen et al., 2023a) treats the LLMs as a black box and
et al., 2023b; Wang et al., 2024c). Chen et al. augments it with a tuneable retrieval model. Ma
(2023b) exploit an NLI model to make entailment et al. (2023b) note that there is inevitably a gap
decisions for each document sentence and answer between the input text and the needed knowledge
sentence pair, then aggregates the results by taking in retrieval. Therefore, to better align the query to
the maximum value over all the pairs. To avoid ob- the retriver, it adopts a small language model as a
taining hallucinated information in the processs of trainable rewriter to cater to the black-box LLM
retrieval, Wang et al. (2024c) utilize an NLI model reader. Howerver, query rewriting lacks the uti-
to determine whether the retrived passages indeed lization of effective and general signals. Therefore,
entail the target information. The QA-driven meth- RaFe (Mao et al., 2024) first trains an initial query
ods apply QA model to obtain the faithfulness of rewriting model by standard supervised fine-tuning,
the generation. Qiu et al. (2023) formulate faith- Subsequently, it utilizes the ranking scores from
fulness evaluation as binary classification problem. the reranker to conduct feedback training on the
Then it develop a classifier to distill knowledge query rewriting model.

16
 For structured data, such as knowledge graph gap in the simultaneous mutual enhancement of
and SQL, it requires additional effort to convert both the servers’ LLM and clients’ SLMs, Fan et al.
structured data to text (Li et al., 2024d; Su et al., (2024) develop FedMKT, a parameter-efficient fed-
2024; Wang et al., 2023a; He et al., 2024). There- erated mutual knowledge transfer framework for
fore, the retriever needs to have the ability to con- large and small language models. Ye et al. (2024)
vert between natural language and structured lan- build a research-friendly framework, called Open-
guage. For knowledge graph, the retriever is an FedLLM, where multiple data owners collabora-
entity linking model to retrieve relative structured tively train a shared model without transmitting raw
knowledge (Su et al., 2024; Wang et al., 2023a). data. Additionally, to minimize resource consump-
Additionally, for SQL knowledge, the retriever is tion of parameters exchange, FedCyBGD (Wang
a query generator that allows the generation of et al., 2024d) design a compression scheme to fur-
queries for various types of query languages, in- ther decrease the model download.
cluding SPARQL, SQL, and natural sentences (Li
et al., 2024d; Zha et al., 2023).

5.4 Federated Cooperation

5.4.2 Federated Prompt Engineering
Data Sharing
The goal of large model collaboration is to protect
user privacy while efficiently executing commands.
Therefore, using local small models to protect user
privacy and cloud large models to execute user com-
mands is a feasible approach (Zhang et al., 2024b;
Li et al., 2024a; Guo et al., 2022). For example,
Zhang et al. (2024b) deploy the small model locally
to convert user instructions into general instruc-
Server LLM Client SLM
tions. The large model in the cloud executes these
Figure 12: LLMs cooperate with client models in feder- general instructions and then returns the results
ated learning. to the local device. Because of discrepancies be-
tween LLMs’ generated data and clients’ domain-
Large-scale language models have received in- specific data, the exsiting methods cannot yield
creasing attention, but they face many challenges in substantial improvements in the domain-specific
real-world applications. These challenges mainly tasks. Therefore, it introduces FDKT framework,
stem from the scarcity of public domain data and which enables LLMs to augment data based on
the need to maintain privacy for private data. To ad- domain-specific few-shot demonstrations (Li et al.,
dress these issues, federated learning (FL) (Li et al., 2024a). Furthermore, FL may cost excessive train-
2020) has emerged as a very promising technology ing time for convergence and produce inaccurate
because it can protect private data while allowing models. PromptFL (Guo et al., 2022) introduces
the sharing of public models. Federated learning prompt learner which updates the prompts instead
can be mainly divided into two parts: federated of the whole model. Therefore, both the local train-
training and federated prompt engineering (Chen ing and the global aggregation can be significantly
et al., 2023a). Existing collaborative studies are accelerated.
distributed across two aspects mentioned above.
In addition to the aspects discussed above, we
5.4.1 Federated Training want to emphasize that cooperation among LLMs
The knowledge of the LLMs on the server can be is a very broad research area. Topics such as RLHF
transferred to the small language models on the and the use of agents also fall under LLM coop-
client, while the unique knowledge of the client eration. As these topics have been extensively re-
can enrich the distribution of the LLMs. Conse- viewed (Kaufmann et al., 2023; Xi et al., 2023; Sun
quently, the above methods greatly expand the pre- et al., 2024; Guo et al., 2024), we do not cover
training corpora for both LLMs and small models them in the current version of our paper but will
(Fan et al., 2024; Ye et al., 2024). To bridge the address them in future work.

17
6 Challenges and Future Directions cluding: efficient computation, knowledge trans-
ferring, compensatory cooperation and federated
6.1 Flexible LLMs Merging Methods
cooperation. Given the remarkable emergent ca-
Current LLM merging methods are typically re- pabilities of LLMs, we believe that a wide range
stricted to models with same architecture and com- of applications can be realized through flexible
patible parameters. However, most open-source and judicious cooperation between different LLMs.
LLMs are heterogeneous, rendering current merg- For example, exploring cross-domain applications,
ing methods ineffective. where LLMs can combine their expertise in various
Interestingly, Xu et al. (2024b) demonstrate that fields will unlock new possibilities. Additionally,
the token embeddings of heterogeneous LLMs can human-centered collaboration is also a promising
be projected into a common space by using over- direction (Ma et al., 2024).
lapping tokens as a bridge. However, this method
cannot be successfully adapted to other parame- 7 Conclusion
ters of LLMs, such as self-attention layers and This work presents a survey of collaboration strate-
feed-forward layers, due to the lack of aligned neu- gies for LLMs, categorized into three aspects:
rons and the complexity of parameter distributions. merging, ensemble, and cooperation. For each
We believe that explore the highly correlated neu- aspect, we provide a detailed classification and an
rons (Singh and Jaggi, 2020; Peña et al., 2023; in-depth review of advanced approaches. We be-
Ainsworth et al., 2023; Jordan et al., 2023; Sto- lieve that the collaboration of LLMs will play an
ica et al., 2024) in diverse LLMs hold significant increasingly important role in future research and
potential and interest. Such advancements could hope that this paper offers valuable insights into
revolutionize model fusion techniques, enhancing future research directions.
their flexibility and practicality.

6.2 Balanced Speed and Performance for

LLM Ensemble
As we discussed in §4.1.4, different LLM ensem-
ble methods have their own strengths and weak-
nesses. Achieving a balance between speed and
performance can be challenging.
LLM ensemble methods employed B EFORE in-
ference typically select a appropriate LLM for cur-
rent example according to the pre-trained routers
(Shnitzer et al., 2023). While these methods im-
prove decoding speed, they are coarse-grained and
do not fully harness the potential of ensemble learn-
ing. In contrast, methods used D URING inference
operate at a finer granularity, achieving ensemble
integration at the token level (Xu et al., 2024b;
Huang et al., 2024c; Yu et al., 2024). Such meth-
ods can alleviate errors at each decoding step via
ensemble, relieving exploration bias (Ranzato et al.,
2015; Xu et al., 2020) and holding greater promise
for enhancing performance. However, these meth-
ods suffer from the slower inference speed. Effec-
tively combining these two strategies may strike a
balance between speed and performance improve-
ments, benefiting practical deployments.

6.3 Broader Applications via Cooperation

This work primarily discusses several objectives
achievable through cooperation among LLMs, in-

18
Contributions References
Jinliang Lu designed the overall architecture of Ahmed M Ahmed, Rafael Rafailov, Stepan Sharkov,
Xuechen Li, and Sanmi Koyejo. 2024. Scalable
this paper and was primarily responsible for §1, §2,
ensembling for mitigating reward overoptimisation.
§4 and §5.2. arXiv preprint arXiv:2406.01013.
Ziliang Pang and Yaochen Zhu are responsible
for the architecture of §3, Merging. Specifically, Samuel Ainsworth, Jonathan Hayase, and Siddhartha
Srinivasa. 2023. Git re-basin: Merging models mod-
Ziliang Pang reviewed the literature in §3.1, §5.1,
ulo permutation symmetries. In The Eleventh Inter-
and Yaochen Zhu reviewed the literature in §3.2. national Conference on Learning Representations.
Min Xiao is responsible for the architecture of
§5 Cooperation and reviewed the literature in §5.3 Takuya Akiba, Makoto Shing, Yujin Tang, Qi Sun, and
David Ha. 2024. Evolutionary optimization of model
and §5.4, primarily categorizing the corresponding
merging recipes. CoRR, abs/2403.13187.
studies based on their objectives.
Jiajun Zhang and Rui Xia led the project, de- Muhammad Asif Ali, Zhengping Li, Shu Yang, Keyuan
signed and optimized the overall architecture of the Cheng, Yang Cao, Tianhao Huang, Lijie Hu, Lu Yu,
and Di Wang. 2024. Prompt-saw: Leveraging
survey, responsible for reviewing the entire work.
relation-aware graphs for textual prompt compres-
If you have suggestions or questions about this sion. Preprint, arXiv:2404.00489.
survey, please contact with us. We are very happy
to hear from you. Rohan Anil, Andrew M Dai, Orhan Firat, Melvin John-
son, Dmitry Lepikhin, Alexandre Passos, Siamak
Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng
Chen, et al. 2023. Palm 2 technical report. arXiv
preprint arXiv:2305.10403.

Zhangir Azerbayev, Hailey Schoelkopf, Keiran Paster,

Marco Dos Santos, Stephen McAleer, Albert Q Jiang,
Jia Deng, Stella Biderman, and Sean Welleck. 2023.
Llemma: An open language model for mathematics.
arXiv preprint arXiv:2310.10631.

Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang,
Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei
Huang, et al. 2023. Qwen technical report. arXiv
preprint arXiv:2309.16609.

Zechen Bai, Pichao Wang, Tianjun Xiao, Tong He,

Zongbo Han, Zheng Zhang, and Mike Zheng Shou.
2024. Hallucination of multimodal large language
models: A survey. Preprint, arXiv:2404.18930.

Gioele Barabucci, Victor Shia, Eugene Chu, Benjamin

Harack, and Nathan Fu. 2024. Combining insights
from multiple large language models improves diag-
nostic accuracy. arXiv preprint arXiv:2402.08806.

Leo Breiman. 1996. Bagging predictors. Machine

learning, 24:123–140.

Tom Brown, Benjamin Mann, Nick Ryder, Melanie

Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind
Neelakantan, Pranav Shyam, Girish Sastry, Amanda
Askell, et al. 2020. Language models are few-shot
learners. Advances in neural information processing
systems, 33:1877–1901.

Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu,

Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi,
Cunxiang Wang, Yidong Wang, et al. 2024. A sur-
vey on evaluation of large language models. ACM
Transactions on Intelligent Systems and Technology,
15(3):1–45.

19
Chao Chen, Kai Liu, Ze Chen, Yi Gu, Yue Wu, Natural Language Processing, pages 11781–11791,
Mingyuan Tao, Zhihang Fu, and Jieping Ye. 2024. Singapore. Association for Computational Linguis-
Inside: Llms’ internal states retain the power of hal- tics.
lucination detection. ArXiv, abs/2402.03744.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and
Chaochao Chen, Xiaohua Feng, Jun Zhou, Jianwei Kristina Toutanova. 2019. BERT: Pre-training of
Yin, and Xiaolin Zheng. 2023a. Federated large deep bidirectional transformers for language under-
language model: A position paper. Preprint, standing. In Proceedings of the 2019 Conference of
arXiv:2307.08925. the North American Chapter of the Association for
Computational Linguistics: Human Language Tech-
Hung-Ting Chen, Fangyuan Xu, Shane A. Arora, and nologies, Volume 1 (Long and Short Papers), pages
Eunsol Choi. 2023b. Understanding retrieval aug- 4171–4186, Minneapolis, Minnesota. Association for
mentation for long-form question answering. ArXiv, Computational Linguistics.
abs/2310.12150.
Jifan Chen, Grace Kim, Aniruddh Sriram, Greg Durrett, Thomas G Dietterich. 2000. Ensemble methods in ma-
and Eunsol Choi. 2023c. Complex claim verifica- chine learning. In International workshop on multi-
tion with evidence retrieved in the wild. Preprint, ple classifier systems, pages 1–15. Springer.
arXiv:2305.11859. Dujian Ding, Ankur Mallick, Chi Wang, Robert Sim,
Lingjiao Chen, Matei Zaharia, and James Zou. 2023d. Subhabrata Mukherjee, Victor Ruhle, Laks VS Laksh-
Frugalgpt: How to use large language models while manan, and Ahmed Hassan Awadallah. 2024. Hybrid
reducing cost and improving performance. arXiv llm: Cost-efficient and quality-aware query routing.
preprint arXiv:2305.05176. arXiv preprint arXiv:2404.14618.

Shouyuan Chen, Sherman Wong, Liangjian Chen, and Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiy-
Yuandong Tian. 2023e. Extending context window ong Wu, Baobao Chang, Xu Sun, Jingjing Xu, and
of large language models via positional interpolation. Zhifang Sui. 2022. A survey on in-context learning.
arXiv preprint arXiv:2306.15595. arXiv preprint arXiv:2301.00234.

Yukang Chen, Shengju Qian, Haotian Tang, Xin Lai, David Eigen, Marc’Aurelio Ranzato, and Ilya Sutskever.
Zhijian Liu, Song Han, and Jiaya Jia. 2023f. Lon- 2013. Learning factored representations in a deep
glora: Efficient fine-tuning of long-context large lan- mixture of experts. arXiv preprint arXiv:1312.4314.
guage models. arXiv preprint arXiv:2309.12307.
Jacob Eisenstein, Chirag Nagpal, Alekh Agarwal, Ah-
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, mad Beirami, Alex D’Amour, DJ Dvijotham, Adam
Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Fisch, Katherine Heller, Stephen Pfohl, Deepak Ra-
Barham, Hyung Won Chung, Charles Sutton, Sebas- machandran, et al. 2023. Helping or herding? reward
tian Gehrmann, et al. 2023. Palm: Scaling language model ensembles mitigate but do not eliminate re-
modeling with pathways. Journal of Machine Learn- ward hacking. arXiv preprint arXiv:2312.09244.
ing Research, 24(240):1–113.
Alexander Fabbri, Chien-Sheng Wu, Wenhao Liu, and
Ronan Collobert, Yoshua Bengio, and Samy Bengio. Caiming Xiong. 2022. QAFactEval: Improved QA-
2003. Scaling large learning problems with hard based factual consistency evaluation for summariza-
parallel mixtures. International Journal of pattern tion. In Proceedings of the 2022 Conference of the
recognition and artificial intelligence, 17(03):349– North American Chapter of the Association for Com-
365. putational Linguistics: Human Language Technolo-
Thomas Coste, Usman Anwar, Robert Kirk, and David gies, pages 2587–2601, Seattle, United States. Asso-
Krueger. 2024. Reward model ensembles help miti- ciation for Computational Linguistics.
gate overoptimization. In The Twelfth International Tobias Falke, Leonardo F. R. Ribeiro, Prasetya Ajie
Conference on Learning Representations. Utama, Ido Dagan, and Iryna Gurevych. 2019. Rank-
Nico Daheim, Thomas Möllenhoff, Edoardo Maria ing generated summaries by correctness: An interest-
Ponti, Iryna Gurevych, and Mohammad Emtiyaz ing but challenging application for natural language
Khan. 2023. Model merging by uncertainty-based inference. In Annual Meeting of the Association for
gradient matching. arXiv preprint arXiv:2310.12808. Computational Linguistics.

Pala Tej Deep, Rishabh Bhardwaj, and Soujanya Po- Tao Fan, Guoqiang Ma, Yan Kang, Hanlin Gu, Lixin
ria. 2024. Della-merging: Reducing interference in Fan, and Qiang Yang. 2024. Fedmkt: Federated mu-
model merging through magnitude-based sampling. tual knowledge transfer for large and small language
Preprint, arXiv:2406.11617. models.

Haikang Deng and Colin Raffel. 2023. Reward- William Fedus, Barret Zoph, and Noam Shazeer. 2022.
augmented decoding: Efficient controlled text gener- Switch transformers: Scaling to trillion parameter
ation with a unidirectional reward model. In Proceed- models with simple and efficient sparsity. Journal of
ings of the 2023 Conference on Empirical Methods in Machine Learning Research, 23(120):1–39.

20
Weizhi Fei, Xueyan Niu, Pingyi Zhou, Lu Hou, Bo Bai, Satya Kesav Gundabathula and Sriram R Kolar. 2024.
Lei Deng, and Wei Han. 2023. Extending context Promptmind team at mediqa-corr 2024: Improving
window of large language models via semantic com- clinical text correction with error categorization and
pression. Preprint, arXiv:2312.09571. llm ensembles. arXiv preprint arXiv:2405.08373.
Yoav Freund and Robert E Schapire. 1997. A decision- Hao Guo, Jiyong Jin, and Bin Liu. 2023. Stochas-
theoretic generalization of on-line learning and an tic weight averaging revisited. Applied Sciences,
application to boosting. Journal of Computer and 13(5):2935.
System Sciences, 55(1):119–139.
Tingchen Fu, Deng Cai, Lemao Liu, Shuming Shi, and Taicheng Guo, Xiuying Chen, Yaqi Wang, Ruidi Chang,
Rui Yan. 2024a. Disperse-then-merge: Pushing the Shichao Pei, Nitesh V Chawla, Olaf Wiest, and Xi-
limits of instruction tuning via alignment tax reduc- angliang Zhang. 2024. Large language model based
tion. arXiv preprint arXiv:2405.13432. multi-agents: A survey of progress and challenges.
arXiv preprint arXiv:2402.01680.
Tingchen Fu, Deng Cai, Lemao Liu, Shuming Shi, and
Rui Yan. 2024b. Disperse-then-merge: Pushing the Tao Guo, Song Guo, Junxiao Wang, and Wenchao Xu.
limits of instruction tuning via alignment tax reduc- 2022. Promptfl: Let federated participants coop-
tion. arXiv preprint arXiv:2405.13432. eratively learn prompts instead of models – feder-
ated learning in age of foundation model. Preprint,
Yao Fu, Hao Peng, Litu Ou, Ashish Sabharwal, and arXiv:2208.11625.
Tushar Khot. 2023. Specializing smaller language
models towards multi-step reasoning. In Inter- Hasan Abed Al Kader Hammoud, Umberto Michieli,
national Conference on Machine Learning, pages Fabio Pizzati, Philip Torr, Adel Bibi, Bernard
10421–10430. PMLR. Ghanem, and Mete Ozay. 2024. Model merging and
Jun Gao. 2024a. Selfcp: Compressing long prompt safety alignment: One bad model spoils the bunch.
to 1/12 using the frozen large language model itself. arXiv preprint arXiv:2406.14563.
Preprint, arXiv:2405.17052.
Xiaoxin He, Yijun Tian, Yifei Sun, Nitesh V. Chawla,
Jun Gao. 2024b. Unifying demonstration selection Thomas Laurent, Yann LeCun, Xavier Bresson, and
and compression for in-context learning. Preprint, Bryan Hooi. 2024. G-retriever: Retrieval-augmented
arXiv:2405.17062. generation for textual graph understanding and ques-
tion answering. Preprint, arXiv:2402.07630.
Luyu Gao, Zhuyun Dai, Panupong Pasupat, Anthony
Chen, Arun Tejasvi Chaganty, Yicheng Fan, Vin- Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015.
cent Zhao, N. Lao, Hongrae Lee, Da-Cheng Juan, Distilling the knowledge in a neural network. arXiv
and Kelvin Guu. 2022. Attributed text genera- preprint arXiv:1503.02531.
tion via post-hoc research and revision. ArXiv,
abs/2210.08726. Hieu Hoang, Huda Khayrallah, and Marcin Junczys-
Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Dowmunt. 2023. On-the-fly fusion of large language
Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, Meng Wang, models and machine translation. arXiv preprint
and Haofen Wang. 2024. Retrieval-augmented gener- arXiv:2311.08306.
ation for large language models: A survey. Preprint,
arXiv:2312.10997. Jordan Hoffmann, Sebastian Borgeaud, Arthur Men-
sch, Elena Buchatskaya, Trevor Cai, Eliza Ruther-
Timur Garipov, Pavel Izmailov, Dmitrii Podoprikhin, ford, Diego de Las Casas, Lisa Anne Hendricks,
Dmitry Vetrov, and Andrew Gordon Wilson. 2018. Johannes Welbl, Aidan Clark, et al. 2022. Train-
Loss surfaces, mode connectivity, and fast ensem- ing compute-optimal large language models. arXiv
bling of dnns. Preprint, arXiv:1802.10026. preprint arXiv:2203.15556.
Charles Goddard, Shamane Siriwardhana, Malikeh Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and
Ehghaghi, Luke Meyers, Vlad Karpukhin, Brian Yejin Choi. 2020a. The curious case of neural text de-
Benedict, Mark McQuade, and Jacob Solawetz. 2024. generation. In International Conference on Learning
Arcee’s mergekit: A toolkit for merging large lan- Representations.
guage models. arXiv preprint arXiv:2403.13257.
Ben Goodrich, Vinay Rao, Mohammad Saleh, and Pe- Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and
ter J. Liu. 2019. Assessing the factual accuracy Yejin Choi. 2020b. The curious case of neural text
of generated text. Proceedings of the 25th ACM degeneration. Preprint, arXiv:1904.09751.
SIGKDD International Conference on Knowledge
Discovery & Data Mining. Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-
Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu
Jiatao Gu, Kyunghyun Cho, and Victor O. K. Li. 2017. Chen. 2022. LoRA: Low-rank adaptation of large
Trainable greedy decoding for neural machine trans- language models. In International Conference on
lation. Preprint, arXiv:1702.02429. Learning Representations.

21
Qitian Jason Hu, Jacob Bieker, Xiuyu Li, Nan Jiang, Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan
Benjamin Keigwin, Gaurav Ranganath, Kurt Keutzer, Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea
and Shriyash Kaustubh Upadhyay. 2024. Router- Madotto, and Pascale Fung. 2023. Survey of halluci-
bench: A benchmark for multi-llm routing system. nation in natural language generation. ACM Comput-
arXiv preprint arXiv:2403.12031. ing Surveys, 55(12):1–38.

Kaixuan Huang, Xudong Guo, and Mengdi Wang. Jianguo Jia, Wen Liang, and Youzhi Liang. 2023. A
2024a. Specdec++: Boosting speculative decod- review of hybrid and ensemble in deep learning
ing via adaptive candidate lengths. arXiv preprint for natural language processing. arXiv preprint
arXiv:2405.19715. arXiv:2312.05589.
Albert Q Jiang, Alexandre Sablayrolles, Antoine
Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Roux, Arthur Mensch, Blanche Savary, Chris Bam-
Zhangyin Feng, Haotian Wang, Qianglong Chen, ford, Devendra Singh Chaplot, Diego de las Casas,
Weihua Peng, Xiaocheng Feng, Bing Qin, et al. 2023. Emma Bou Hanna, Florian Bressand, et al. 2024.
A survey on hallucination in large language models: Mixtral of experts. arXiv preprint arXiv:2401.04088.
Principles, taxonomy, challenges, and open questions.
arXiv preprint arXiv:2311.05232. Dongfu Jiang, Xiang Ren, and Bill Yuchen Lin. 2023a.
Llm-blender: Ensembling large language models
Xijie Huang, Li Lyna Zhang, Kwang-Ting Cheng, Fan with pairwise ranking and generative fusion. arXiv
Yang, and Mao Yang. 2024b. Fewer is more: Boost- preprint arXiv:2306.02561.
ing llm reasoning with reinforced context pruning.
Preprint, arXiv:2312.08901. Huiqiang Jiang, Qianhui Wu, Chin-Yew Lin, Yuqing
Yang, and Lili Qiu. 2023b. LLMLingua: Compress-
Yichong Huang, Xiaocheng Feng, Baohang Li, Yang ing prompts for accelerated inference of large lan-
Xiang, Hui Wang, Bing Qin, and Ting Liu. 2024c. guage models. In Proceedings of the 2023 Confer-
Enabling ensemble learning for heterogeneous large ence on Empirical Methods in Natural Language Pro-
language models with deep parallel collaboration. cessing, pages 13358–13376, Singapore. Association
arXiv preprint arXiv:2404.12715. for Computational Linguistics.
Xisen Jin, Xiang Ren, Daniel Preotiuc-Pietro, and
Siqing Huo, Negar Arabzadeh, and Charles L. A. Clarke.
Pengxiang Cheng. 2022. Dataless knowledge fu-
2023. Retrieving supporting evidence for llms gener-
sion by merging weights of language models. arXiv
ated answers. ArXiv, abs/2306.13781.
preprint arXiv:2212.09849.
Gabriel Ilharco, Marco Túlio Ribeiro, Mitchell Worts- Keller Jordan, Hanie Sedghi, Olga Saukh, Rahim En-
man, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali tezari, and Behnam Neyshabur. 2023. REPAIR:
Farhadi. 2023. Editing models with task arithmetic. REnormalizing permuted activations for interpola-
In The Eleventh International Conference on Learn- tion repair. In The Eleventh International Conference
ing Representations, ICLR 2023, Kigali, Rwanda, on Learning Representations.
May 1-5, 2023. [Link].
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B
Gautier Izacard, Mathilde Caron, Lucas Hosseini, Se- Brown, Benjamin Chess, Rewon Child, Scott Gray,
bastian Riedel, Piotr Bojanowski, Armand Joulin, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020.
and Edouard Grave. 2021. Unsupervised dense in- Scaling laws for neural language models. arXiv
formation retrieval with contrastive learning. Trans. preprint arXiv:2001.08361.
Mach. Learn. Res., 2022.
Timo Kaufmann, Paul Weng, Viktor Bengs, and Eyke
Gautier Izacard, Patrick Lewis, Maria Lomeli, Lucas Hüllermeier. 2023. A survey of reinforcement
Hosseini, Fabio Petroni, Timo Schick, Jane A. Yu, learning from human feedback. arXiv preprint
Armand Joulin, Sebastian Riedel, and Edouard Grave. arXiv:2312.14925.
2022. Few-shot learning with retrieval augmented Seungone Kim, Juyoung Suk, Shayne Longpre,
language models. ArXiv, abs/2208.03299. Bill Yuchen Lin, Jamin Shin, Sean Welleck, Graham
Neubig, Moontae Lee, Kyungjae Lee, and Minjoon
Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Seo. 2024. Prometheus 2: An open source language
Dmitry Vetrov, and Andrew Gordon Wilson. 2018. model specialized in evaluating other language mod-
Averaging weights leads to wider optima and better els. arXiv preprint arXiv:2405.01535.
generalization. arXiv preprint arXiv:1803.05407.
Ben Krause, Akhilesh Deepak Gotmare, Bryan McCann,
Robert A Jacobs, Michael I Jordan, Steven J Nowlan, Nitish Shirish Keskar, Shafiq Joty, Richard Socher,
and Geoffrey E Hinton. 1991. Adaptive mixtures of and Nazneen Fatema Rajani. 2021. GeDi: Gener-
local experts. Neural computation, 3(1):79–87. ative discriminator guided sequence generation. In
Findings of the Association for Computational Lin-
Dong-Hwan Jang, Sangdoo Yun, and Dongyoon Han. guistics: EMNLP 2021, pages 4929–4952, Punta
2024. Model stock: All we need is just a few fine- Cana, Dominican Republic. Association for Compu-
tuned models. ArXiv, abs/2403.19522. tational Linguistics.

22
Teven Le Scao, Angela Fan, Christopher Akiki, El- Xingxuan Li, Ruochen Zhao, Yew Ken Chia, Bosheng
lie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Ding, Shafiq Joty, Soujanya Poria, and Lidong Bing.
Castagné, Alexandra Sasha Luccioni, François Yvon, 2024d. Chain-of-knowledge: Grounding large lan-
Matthias Gallé, et al. 2023. Bloom: A 176b- guage models via dynamic knowledge adapting over
parameter open-access multilingual language model. heterogeneous sources. In The Twelfth International
Conference on Learning Representations.
Young-Suk Lee, Md Arafat Sultan, Yousef El-Kurdi,
Tahira Naseem Asim Munawar, Radu Florian, Salim Xi Victoria Lin, Todor Mihaylov, Mikel Artetxe, Tianlu
Roukos, and Ramón Fernandez Astudillo. 2023. Wang, Shuohui Chen, Daniel Simig, Myle Ott, Na-
Ensemble-instruct: Generating instruction-tuning man Goyal, Shruti Bhosale, Jingfei Du, Ramakanth
data with a heterogeneous mixture of lms. arXiv Pasunuru, Sam Shleifer, Punit Singh Koura, Vishrav
preprint arXiv:2310.13961. Chaudhary, Brian O’Horo, Jeff Wang, Luke Zettle-
moyer, Zornitsa Kozareva, Mona Diab, Veselin Stoy-
Yaniv Leviathan, Matan Kalman, and Yossi Matias. anov, and Xian Li. 2022. Few-shot learning with
2023. Fast inference from transformers via spec- multilingual generative language models. In Proceed-
ulative decoding. In International Conference on ings of the 2022 Conference on Empirical Methods
Machine Learning, pages 19274–19286. PMLR. in Natural Language Processing, pages 9019–9052,
Abu Dhabi, United Arab Emirates. Association for
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Computational Linguistics.
Ghazvininejad, Abdelrahman Mohamed, Omer Levy,
Veselin Stoyanov, and Luke Zettlemoyer. 2020. Yong Lin, Lu Tan, Hangyu Lin, Zeming Zheng, Renjie
BART: Denoising sequence-to-sequence pre-training Pi, Jipeng Zhang, Shizhe Diao, Haoxiang Wang, Han
for natural language generation, translation, and com- Zhao, Yuan Yao, and T. Zhang. 2023. Mitigating the
prehension. In Proceedings of the 58th Annual Meet- alignment tax of rlhf.
ing of the Association for Computational Linguistics,
pages 7871–7880, Online. Association for Computa- Alisa Liu, Xiaochuang Han, Yizhong Wang, Yulia
tional Linguistics. Tsvetkov, Yejin Choi, and Noah A Smith. 2024a.
Tuning language models by proxy. arXiv preprint
Haoran Li, Xinyuan Zhao, Dadi Guo, Hanlin Gu, Ziqian arXiv:2401.08565.
Zeng, Yuxing Han, Yangqiu Song, Lixin Fan, and
Alisa Liu, Maarten Sap, Ximing Lu, Swabha
Qiang Yang. 2024a. Federated domain-specific
Swayamdipta, Chandra Bhagavatula, Noah A Smith,
knowledge transfer on large language models using
and Yejin Choi. 2021. Dexperts: Decoding-time con-
synthetic data.
trolled text generation with experts and anti-experts.
Jinyi Li, Yihuai Lan, Lei Wang, and Hao Wang. 2024b. In Proceedings of the 59th Annual Meeting of the As-
Pctoolkit: A unified plug-and-play prompt compres- sociation for Computational Linguistics and the 11th
sion toolkit of large language models. Preprint, International Joint Conference on Natural Language
arXiv:2403.17411. Processing (Volume 1: Long Papers).
Deyuan Liu, Zecheng Wang, Bingning Wang, Weipeng
Margaret Li, Suchin Gururangan, Tim Dettmers, Mike
Chen, Chunshan Li, Zhiying Tu, Dianhui Chu, Bo Li,
Lewis, Tim Althoff, Noah A Smith, and Luke Zettle-
and Dianbo Sui. 2024b. Checkpoint merging via
moyer. 2022. Branch-train-merge: Embarrassingly
bayesian optimization in llm pretraining. arXiv
parallel training of expert language models. arXiv
preprint arXiv:2403.19390.
preprint arXiv:2208.03306.
Fangcheng Liu, Yehui Tang, Zhenhua Liu, Yunsheng
Tian Li, Anit Kumar Sahu, Ameet Talwalkar, and Vir- Ni, Kai Han, and Yunhe Wang. 2024c. Kangaroo:
ginia Smith. 2020. Federated learning: Challenges, Lossless self-speculative decoding via double early
methods, and future directions. IEEE Signal Process- exiting. arXiv preprint arXiv:2404.18911.
ing Magazine, 37(3):50–60.
Jiahao Liu, Qifan Wang, Jingang Wang, and Xunliang
Tianlin Li, Qian Liu, Tianyu Pang, Chao Du, Qing Guo, Cai. 2024d. Speculative decoding via early-exiting
Yang Liu, and Min Lin. 2024c. Purifying large lan- for faster llm inference with thompson sampling con-
guage models by ensembling a small language model. trol mechanism. arXiv preprint arXiv:2406.03853.
arXiv preprint arXiv:2402.14845.
Junyi Liu, Liangzhi Li, Tong Xiang, Bowen Wang, and
Weishi Li, Yong Peng, Miao Zhang, Liang Ding, Han Yiming Qian. 2023. TCRA-LLM: Token compres-
Hu, and Li Shen. 2023a. Deep model fusion: A sion retrieval augmented large language model for
survey. arXiv preprint arXiv:2309.15698. inference cost reduction. In Findings of the Associ-
ation for Computational Linguistics: EMNLP 2023,
Xiang Lisa Li, Ari Holtzman, Daniel Fried, Percy Liang, pages 9796–9810, Singapore. Association for Com-
Jason Eisner, Tatsunori B Hashimoto, Luke Zettle- putational Linguistics.
moyer, and Mike Lewis. 2023b. Contrastive decod-
ing: Open-ended text generation as optimization. In Jinliang Lu, Chen Wang, and Jiajun Zhang. 2024a.
Proceedings of the 61st Annual Meeting of the As- Diver: Large language model decoding with span-
sociation for Computational Linguistics (Volume 1: level mutual information verification. arXiv preprint
Long Papers), pages 12286–12312. arXiv:2406.02120.

23
Keming Lu, Bowen Yu, Fei Huang, Yang Fan, Runji Lin, Bonan Min, Hayley Ross, Elior Sulem, Amir
and Chang Zhou. 2024b. Online merging optimizers Pouran Ben Veyseh, Thien Huu Nguyen, Oscar Sainz,
for boosting rewards and mitigating tax in alignment. Eneko Agirre, Ilana Heintz, and Dan Roth. 2023a.
arXiv preprint arXiv:2405.17931. Recent advances in natural language processing via
large pre-trained language models: A survey. ACM
Keming Lu, Hongyi Yuan, Runji Lin, Junyang Lin, Computing Surveys, 56(2):1–40.
Zheng Yuan, Chang Zhou, and Jingren Zhou. 2023.
Routing to the expert: Efficient reward-guided en- Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike
semble of large language models. arXiv preprint Lewis, Wen tau Yih, Pang Wei Koh, Mohit Iyyer,
arXiv:2311.08692. Luke Zettlemoyer, and Hannaneh Hajishirzi. 2023b.
Factscore: Fine-grained atomic evaluation of fac-
Xiaoding Lu, Adian Liusie, Vyas Raina, Yuwen Zhang, tual precision in long form text generation. Preprint,
and William Beauchamp. 2024c. Blending is all arXiv:2305.14251.
you need: Cheaper, better alternative to trillion-
parameters llm. arXiv preprint arXiv:2401.02994. Anshuman Mishra, Dhruvesh Patel, Aparna Vijayaku-
mar, Xiang Lorraine Li, Pavan Kapanipathi, and Kar-
Xinbei Ma, Yeyun Gong, Pengcheng He, Hai Zhao, tik Talamadupula. 2021. Looking beyond sentence-
and Nan Duan. 2023a. Query rewriting for level natural language inference for question answer-
retrieval-augmented large language models. ArXiv, ing and text summarization. In North American
abs/2305.14283. Chapter of the Association for Computational Lin-
guistics.
Xinbei Ma, Yeyun Gong, Pengcheng He, Hai Zhao,
and Nan Duan. 2023b. Query rewriting in retrieval- Alireza Mohammadshahi, Ali Shaikh, and Majid Yaz-
augmented large language models. In Proceedings of dani. 2024. Leeroo orchestrator: Elevating llms per-
the 2023 Conference on Empirical Methods in Natu- formance through model integration. arXiv preprint
ral Language Processing, pages 5303–5315, Singa- arXiv:2401.13979.
pore. Association for Computational Linguistics.
Feng Nan, Ramesh Nallapati, Zhiguo Wang,
Yongqiang Ma, Lizhi Qin, Jiawei Liu, Yangyang Kang, Cícero Nogueira dos Santos, Henghui Zhu,
Yue Zhang, Wei Lu, Xiaozhong Liu, and Qikai Dejiao Zhang, Kathleen McKeown, and Bing Xiang.
Cheng. 2024. From model-centered to human- 2021. Entity-level factual consistency of abstractive
centered: Revision distance as a metric for text eval- text summarization. In Conference of the European
uation in llms-based applications. arXiv preprint Chapter of the Association for Computational
arXiv:2404.07108. Linguistics.
Aman Madaan, Pranjal Aggarwal, Ankit Anand, Sriv-
Ganesh Nathan et al. 2024. Fisher mask nodes
idya Pranavi Potharaju, Swaroop Mishra, Pei Zhou,
for language model merging. arXiv preprint
Aditya Gupta, Dheeraj Rajagopal, Karthik Kappa-
arXiv:2403.09891.
ganthu, Yiming Yang, et al. 2023. Automix: Auto-
matically mixing language models. arXiv preprint
arXiv:2310.12963. Tong Niu, Caiming Xiong, Semih Yavuz, and
Yingbo Zhou. 2024. Parameter-efficient detoxifi-
Shengyu Mao, Yong Jiang, Boli Chen, Xiao Li, Peng cation with contrastive decoding. arXiv preprint
Wang, Xinyu Wang, Pengjun Xie, Fei Huang, Huajun arXiv:2401.06947.
Chen, and Ningyu Zhang. 2024. Rafe: Ranking
feedback improves query rewriting for rag. ArXiv, Sean O’Brien and Mike Lewis. 2023. Contrastive de-
abs/2405.14431. coding improves reasoning in large language models.
arXiv preprint arXiv:2309.09117.
Michael S Matena and Colin A Raffel. 2022. Merging
models with fisher-weighted averaging. Advances in David Oniani, Jordan Hilsman, Hang Dong, Fengyi
Neural Information Processing Systems, 35:17703– Gao, Shiven Verma, and Yanshan Wang. 2023. Large
17716. language models vote: Prompting for rare disease
identification. arXiv preprint arXiv:2308.12890.
Costas Mavromatis, Petros Karypis, and George
Karypis. 2024. Pack of llms: Model fusion at test- Aitor Ormazabal, Mikel Artetxe, and Eneko Agirre.
time via perplexity optimization. arXiv preprint 2023. CombLM: Adapting black-box language mod-
arXiv:2404.11531. els through small fine-tuned models. In Proceedings
of the 2023 Conference on Empirical Methods in Nat-
Xupeng Miao, Gabriele Oliaro, Zhihao Zhang, Xinhao ural Language Processing, pages 2961–2974, Singa-
Cheng, Zeyu Wang, Rae Ying Yee Wong, Zhuom- pore. Association for Computational Linguistics.
ing Chen, Daiyaan Arfeen, Reyna Abhyankar, and
Zhihao Jia. 2023. Specinfer: Accelerating generative Jie Ou, Yueming Chen, and Wenhong Tian. 2024. Loss-
llm serving with speculative inference and token tree less acceleration of large language model via adaptive
verification. ArXiv, abs/2305.09781. n-gram parallel decoding. ArXiv, abs/2404.08698.

24
Nedjma Djouhra Ousidhoum, Moy Yuan, and Andreas Alexandre Rame, Matthieu Kirchmeyer, Thibaud
Vlachos. 2022. Varifocal question generation for Rahier, Alain Rakotomamonjy, Patrick Gallinari, and
fact-checking. In Conference on Empirical Methods Matthieu Cord. 2022. Diverse weight averaging
in Natural Language Processing. for out-of-distribution generalization. Advances in
Neural Information Processing Systems, 35:10821–
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, 10836.
Carroll Wainwright, Pamela Mishkin, Chong Zhang,
Sandhini Agarwal, Katarina Slama, Alex Ray, John Alexandre Ram’e, Nino Vieillard, L’eonard Hussenot,
Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Robert Dadashi, Geoffrey Cideron, Olivier Bachem,
Maddie Simens, Amanda Askell, Peter Welinder, and Johan Ferret. 2024. Warm: On the bene-
Paul F Christiano, Jan Leike, and Ryan Lowe. 2022. fits of weight averaged reward models. ArXiv,
Training language models to follow instructions with abs/2401.12187.
human feedback. In Advances in Neural Information
Processing Systems, volume 35, pages 27730–27744. Alexandre Ramé, Nino Vieillard, Léonard Hussenot,
Curran Associates, Inc. Robert Dadashi, Geoffrey Cideron, Olivier Bachem,
and Johan Ferret. 2024. Warm: On the benefits
Liangming Pan, Xiaobao Wu, Xinyuan Lu, Anh Tuan of weight averaged reward models. arXiv preprint
Luu, William Yang Wang, Min-Yen Kan, and Preslav arXiv:2401.12187.
Nakov. 2023. Fact-checking complex claims with
program-guided reasoning. ArXiv, abs/2305.12744. Marc’Aurelio Ranzato, Sumit Chopra, Michael Auli,
and Wojciech Zaremba. 2015. Sequence level train-
Zhuoshi Pan, Qianhui Wu, Huiqiang Jiang, Menglin Xia, ing with recurrent neural networks. arXiv preprint
Xufang Luo, Jue Zhang, Qingwei Lin, Victor Rühle, arXiv:1511.06732.
Yuqing Yang, Chin-Yew Lin, H. Vicky Zhao, Lili Qiu,
and Dongmei Zhang. 2024. Llmlingua-2: Data distil- Marc’Aurelio Ranzato, Sumit Chopra, Michael Auli,
lation for efficient and faithful task-agnostic prompt and Wojciech Zaremba. 2016. Sequence level train-
compression. Preprint, arXiv:2403.12968. ing with recurrent neural networks. In The Sixth
International Conference on Learning Representa-
Liviu Panait and Sean Luke. 2005. Cooperative multi- tions.
agent learning: The state of the art. Autonomous
agents and multi-agent systems, 11:387–434. Vipula Rawte, Amit Sheth, and Amitava Das. 2023. A
survey of hallucination in large foundation models.
Fidel A Guerrero Peña, Heitor Rapela Medeiros, arXiv preprint arXiv:2309.05922.
Thomas Dubail, Masih Aminbeidokhti, Eric Granger,
and Marco Pedersoli. 2023. Re-basin via implicit Clemens Rosenbaum, Tim Klinger, and Matthew
sinkhorn differentiation. In Proceedings of the Riemer. 2017. Routing networks: Adaptive selec-
IEEE/CVF Conference on Computer Vision and Pat- tion of non-linear functions for multi-task learning.
tern Recognition, pages 20237–20246. arXiv preprint arXiv:1711.01239.

Phuc Phan, Hieu Tran, and Long Phan. 2024. Distil- Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten
lation contrastive decoding: Improving llms reason- Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi,
ing with contrastive decoding and distillation. arXiv Jingyu Liu, Tal Remez, Jérémy Rapin, et al. 2023.
preprint arXiv:2402.14874. Code llama: Open foundation models for code. arXiv
preprint arXiv:2308.12950.
Yifu Qiu, Yftah Ziser, Anna Korhonen, Edoardo Ponti,
and Shay Cohen. 2023. Detecting and mitigating Omer Sagi and Lior Rokach. 2018. Ensemble learn-
hallucinations in multilingual summarisation. In Pro- ing: A survey. Wiley interdisciplinary reviews: data
ceedings of the 2023 Conference on Empirical Meth- mining and knowledge discovery, 8(4):e1249.
ods in Natural Language Processing, pages 8914–
8932, Singapore. Association for Computational Lin- Victor Sanh, Albert Webson, Colin Raffel, Stephen H
guistics. Bach, Lintang Sutawika, Zaid Alyafeai, Antoine
Chaffin, Arnaud Stiegler, Teven Le Scao, Arun
Fanyi Qu, Hao Sun, and Yunfang Wu. 2024. Unsuper- Raja, et al. 2021. Multitask prompted training en-
vised distractor generation via large language model ables zero-shot task generalization. arXiv preprint
distilling and counterfactual contrastive decoding. arXiv:2110.08207.
arXiv preprint arXiv:2406.01306.
Thibault Sellam, Dipanjan Das, and Ankur Parikh. 2020.
Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Bleurt: Learning robust metrics for text generation.
Sutskever, et al. 2018. Improving language under- In Proceedings of the 58th Annual Meeting of the As-
standing by generative pre-training. sociation for Computational Linguistics, pages 7881–
7892.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine
Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Chufan Shi, Cheng Yang, Xinyu Zhu, Jiahao Wang,
Wei Li, and Peter J Liu. 2020. Exploring the lim- Taiqiang Wu, Siheng Li, Deng Cai, Yujiu Yang, and
its of transfer learning with a unified text-to-text Yu Meng. 2024. Unchosen experts can contribute
transformer. Journal of machine learning research, too: Unleashing moe models’ power by self-contrast.
21(140):1–67. arXiv preprint arXiv:2405.14507.

25
Tal Shnitzer, Anthony Ou, Mírian Silva, Kate Soule, Hugo Touvron, Louis Martin, Kevin Stone, Peter Al-
Yuekai Sun, Justin Solomon, Neil Thompson, and bert, Amjad Almahairi, Yasmine Babaei, Nikolay
Mikhail Yurochkin. 2023. Large language model Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti
routing with benchmark datasets. arXiv preprint Bhosale, et al. 2023b. Llama 2: Open founda-
arXiv:2309.15789. tion and fine-tuned chat models. arXiv preprint
arXiv:2307.09288.
Chenglei Si, Weijia Shi, Chen Zhao, Luke Zettlemoyer,
and Jordan Boyd-Graber. 2023. Getting more out Yao-Hung Tsai, Walter Talbott, and Jian Zhang. 2024.
of mixture of language model reasoning experts. In Efficient non-parametric uncertainty quantification
Findings of the Association for Computational Lin- for black-box large language models and decision
guistics: EMNLP 2023, pages 8234–8249. planning. ArXiv, abs/2402.00251.
Lifu Tu, Semih Yavuz, Jin Qu, Jiacheng Xu, Rui Meng,
Sidak Pal Singh and Martin Jaggi. 2020. Model fusion Caiming Xiong, and Yingbo Zhou. 2023. Unlocking
via optimal transport. Advances in Neural Informa- anticipatory text generation: A constrained approach
tion Processing Systems, 33:22045–22055. for faithful decoding with large language models.
arXiv preprint arXiv:2312.06149.
KV Srivatsa, Kaushal Kumar Maurya, and Ekaterina
Kochmar. 2024. Harnessing the power of multiple Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob
minds: Lessons learned from llm routing. arXiv Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz
preprint arXiv:2405.00467. Kaiser, and Illia Polosukhin. 2017. Attention is all
you need. Advances in neural information processing
Mitchell Stern, Noam Shazeer, and Jakob Uszkoreit. systems, 30.
2018. Blockwise parallel decoding for deep autore-
gressive models. In Advances in Neural Information Jonas Waldendorf, Barry Haddow, and Alexandra Birch.
Processing Systems, volume 31. Curran Associates, 2024. Contrastive decoding reduces hallucinations
Inc. in large multilingual machine translation models. In
Proceedings of the 18th Conference of the European
George Stoica, Daniel Bolya, Jakob Bjorner, Pratik Chapter of the Association for Computational Lin-
Ramesh, Taylor Hearn, and Judy Hoffman. 2024. guistics (Volume 1: Long Papers), pages 2526–2539.
Zipit! merging models from different tasks without
training. Preprint, arXiv:2305.03053. Fanqi Wan, Xinting Huang, Deng Cai, Xiaojun Quan,
Wei Bi, and Shuming Shi. 2024a. Knowledge fusion
of large language models. In The Twelfth Interna-
Xin Su, Tiep Le, Steven Bethard, and Phillip Howard. tional Conference on Learning Representations.
2024. Semi-structured chain-of-thought: Integrat-
ing multiple sources of knowledge for improved lan- Fanqi Wan, Ziyi Yang, Longguang Zhong, Xiaojun
guage model reasoning. Preprint, arXiv:2311.08505. Quan, Xinting Huang, and Wei Bi. 2024b. Fusechat:
Knowledge fusion of chat models. arXiv preprint
Chuanneng Sun, Songjun Huang, and Dario Pompili. arXiv:2402.16107.
2024. Llm-based multi-agent reinforcement learn-
ing: Current and future directions. arXiv preprint Zhongwei Wan, Xin Wang, Che Liu, Samiul Alam,
arXiv:2405.11106. Yu Zheng, Jiachen Liu, Zhongnan Qu, Shen Yan,
Yi Zhu, Quanlu Zhang, Mosharaf Chowdhury, and
Anke Tang, Li Shen, Yong Luo, Liang Ding, Han Hu, Mi Zhang. 2024c. Efficient large language models:
Bo Du, and Dacheng Tao. 2023. Concrete subspace A survey. Preprint, arXiv:2312.03863.
learning based interference elimination for multi-task
model fusion. CoRR, abs/2312.06173. Boxin Wang, Weixin Chen, Hengzhi Pei, Chulin Xie,
Mintong Kang, Chenhui Zhang, Chejian Xu, Zidi
Liyan Tang, Philippe Laban, and Greg Durrett. 2024. Xiong, Ritik Dutta, Rylan Schaeffer, et al. 2024a.
Minicheck: Efficient fact-checking of llms on ground- Decodingtrust: A comprehensive assessment of trust-
ing documents. ArXiv, abs/2404.10774. worthiness in gpt models. Advances in Neural Infor-
mation Processing Systems, 36.
Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Danqing Wang and Lei Li. 2023. Learning from mis-
Shazeer, Apoorv Kulshreshtha, Heng-Tze Cheng, takes via cooperative study assistant for large lan-
Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, et al. guage models. In Proceedings of the 2023 Confer-
2022. Lamda: Language models for dialog applica- ence on Empirical Methods in Natural Language Pro-
tions. arXiv preprint arXiv:2201.08239. cessing, pages 10667–10685, Singapore. Association
for Computational Linguistics.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier
Martinet, Marie-Anne Lachaux, Timothée Lacroix, Hongyi Wang, Felipe Maia Polo, Yuekai Sun, Souvik
Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Kundu, Eric Xing, and Mikhail Yurochkin. 2024b.
Azhar, et al. 2023a. Llama: Open and effi- Fusing models with complementary expertise. In
cient foundation language models. arXiv preprint The Twelfth International Conference on Learning
arXiv:2302.13971. Representations.

26
Keheng Wang, Feiyu Duan, Peiguang Li, Sirui Wang, toward building open-source language models for
and Xunliang Cai. 2024c. Llms know what they need: medicine. Journal of the American Medical Infor-
Leveraging a missing information guided framework matics Association, page ocae045.
to empower retrieval-augmented generation. ArXiv,
abs/2404.14043. Jiayang Wu, Wensheng Gan, Zefeng Chen, Shicheng
Wan, and Philip S. Yu. 2023a. Multimodal large
Lin Wang, Zhichao Wang, and Xiaoying Tang. 2024d. language models: A survey. In 2023 IEEE Inter-
Save it all: Enabling full parameter tuning for feder- national Conference on Big Data (BigData), pages
ated large language models via cycle black gradient 2247–2256.
descent.
Shijie Wu, Ozan Irsoy, Steven Lu, Vadim Dabravolski,
Xintao Wang, Qianwen Yang, Yongting Qiu, Jiaqing Mark Dredze, Sebastian Gehrmann, Prabhanjan Kam-
Liang, Qianyu He, Zhouhong Gu, Yanghua Xiao, and badur, David Rosenberg, and Gideon Mann. 2023b.
Wei Wang. 2023a. Knowledgpt: Enhancing large Bloomberggpt: A large language model for finance.
language models with retrieval and storage access on arXiv preprint arXiv:2303.17564.
knowledge bases. Preprint, arXiv:2308.11761.
Zhiheng Xi, Wenxiang Chen, Xin Guo, Wei He, Yiwen
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Ding, Boyang Hong, Ming Zhang, Junzhe Wang,
Le, Ed Chi, and Denny Zhou. 2022. Rationale- Senjie Jin, Enyu Zhou, et al. 2023. The rise and
augmented ensembles in language models. arXiv potential of large language model based agents: A
preprint arXiv:2207.00747. survey. arXiv preprint arXiv:2309.07864.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Heming Xia, Tao Ge, Peiyi Wang, Si-Qing Chen, Furu
Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, Wei, and Zhifang Sui. 2023. Speculative decod-
and Denny Zhou. 2023b. Self-consistency improves ing: Exploiting speculative execution for accelerat-
chain of thought reasoning in language models. In ing seq2seq generation. In Findings of the Associa-
The Eleventh International Conference on Learning tion for Computational Linguistics: EMNLP 2023,
Representations. pages 3909–3925, Singapore. Association for Com-
putational Linguistics.
Yuxia Wang, Revanth Gangi Reddy, Zain Muhammad
Mujahid, Arnav Arora, Aleksandr Rubashevskii, Ji- Heming Xia, Zhe Yang, Qingxiu Dong, Peiyi Wang,
ahui Geng, Osama Mohammed Afzal, Liangming Yongqi Li, Tao Ge, Tianyu Liu, Wenjie Li, and Zhi-
Pan, Nadav Borenstein, Aditya Pillai, Isabelle Au- fang Sui. 2024. Unlocking efficiency in large lan-
genstein, Iryna Gurevych, and Preslav Nakov. 2023c. guage model inference: A comprehensive survey of
Factcheck-bench: Fine-grained evaluation bench- speculative decoding. Preprint, arXiv:2401.07851.
mark for automatic fact-checkers.
Fangyuan Xu, Weijia Shi, and Eunsol Choi. 2023.
Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Recomp: Improving retrieval-augmented lms with
Adams Wei Yu, Brian Lester, Nan Du, Andrew M. compression and selective augmentation. Preprint,
Dai, and Quoc V Le. 2022a. Finetuned language arXiv:2310.04408.
models are zero-shot learners. In International Con-
ference on Learning Representations. Mengwei Xu, Wangsong Yin, Dongqi Cai, Rongjie
Yi, Daliang Xu, Qipeng Wang, Bingyang Wu,
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Yihao Zhao, Chen Yang, Shihe Wang, Qiyang
Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Zhang, Zhenyan Lu, Li Zhang, Shangguang Wang,
Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Yuanchun Li, Yunxin Liu, Xin Jin, and Xuanzhe
Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liu. 2024a. A survey of resource-efficient llm
Liang, Jeff Dean, and William Fedus. 2022b. Emer- and multimodal foundation models. Preprint,
gent abilities of large language models. Transactions arXiv:2401.08092.
on Machine Learning Research. Survey Certifica-
tion. Yangyifan Xu, Jinliang Lu, and Jiajun Zhang. 2024b.
Bridging the gap between different vocabularies for
David H Wolpert. 1992. Stacked generalization. Neural llm ensemble. arXiv preprint arXiv:2404.09492.
networks, 5(2):241–259.
Yifan Xu, Kening Zhang, Haoyu Dong, Yuezhou Sun,
Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Wenlong Zhao, and Tu Zhuowen. 2020. Rethink-
Rebecca Roelofs, Raphael Gontijo-Lopes, Ari S Mor- ing exposure bias in adversarial language modeling.
cos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, arXiv preprint arXiv:1910.11235.
Simon Kornblith, et al. 2022. Model soups: averag-
ing weights of multiple fine-tuned models improves Prateek Yadav, Derek Tam, Leshem Choshen, Colin A.
accuracy without increasing inference time. In In- Raffel, and Mohit Bansal. 2023. Ties-merging: Re-
ternational conference on machine learning, pages solving interference when merging models. In Ad-
23965–23998. PMLR. vances in Neural Information Processing Systems 36:
Annual Conference on Neural Information Process-
Chaoyi Wu, Weixiong Lin, Xiaoman Zhang, Ya Zhang, ing Systems 2023, NeurIPS 2023, New Orleans, LA,
Weidi Xie, and Yanfeng Wang. 2024. Pmc-llama: USA, December 10 - 16, 2023.

27
Enneng Yang, Li Shen, Zhenyi Wang, Guibing Guo, Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang,
Xiaojun Chen, Xingwei Wang, and Dacheng Tao. Hanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu,
2024a. Representation surgery for multi-task model Wendi Zheng, Xiao Xia, Weng Lam Tam, Zixuan Ma,
merging. CoRR, abs/2402.02705. Yufei Xue, Jidong Zhai, Wenguang Chen, Zhiyuan
Liu, Peng Zhang, Yuxiao Dong, and Jie Tang. 2023.
Enneng Yang, Zhenyi Wang, Li Shen, Shiwei Liu, Guib- GLM-130b: An open bilingual pre-trained model. In
ing Guo, Xingwei Wang, and Dacheng Tao. 2023. The Eleventh International Conference on Learning
Adamerging: Adaptive model merging for multi-task Representations.
learning. CoRR, abs/2310.02575.
Liangyu Zha, Junlin Zhou, Liyao Li, Rui Wang, Qingyi
Kevin Yang and Dan Klein. 2021. FUDGE: Controlled Huang, Saisai Yang, Jing Yuan, Changbao Su, Xi-
text generation with future discriminators. In Pro- ang Li, Aofeng Su, Tao Zhang, Chen Zhou, Kaizhe
ceedings of the 2021 Conference of the North Amer- Shou, Miao Wang, Wufang Zhu, Guoshan Lu, Chao
ican Chapter of the Association for Computational Ye, Yali Ye, Wentao Ye, Yiming Zhang, Xinglong
Linguistics: Human Language Technologies, pages Deng, Jie Xu, Haobo Wang, Gang Chen, and Junbo
3511–3535, Online. Association for Computational Zhao. 2023. Tablegpt: Towards unifying tables, na-
Linguistics. ture language and commands into one gpt. Preprint,
arXiv:2307.08674.
Songhua Yang, Hanjie Zhao, Senbin Zhu, Guangyu
Zhou, Hongfei Xu, Yuxiang Jia, and Hongying Zan. Yuanzhao Zhai, Han Zhang, Yu Lei, Yue Yu, Kele Xu,
2024b. Zhongjing: Enhancing the chinese medical Dawei Feng, Bo Ding, and Huaimin Wang. 2023.
capabilities of large language model through expert Uncertainty-penalized reinforcement learning from
feedback and real-world multi-turn dialogue. In Pro- human feedback with diverse reward lora ensembles.
ceedings of the AAAI Conference on Artificial Intelli- arXiv preprint arXiv:2401.00243.
gence, volume 38, pages 19368–19376.
Biao Zhang, Behrooz Ghorbani, Ankur Bapna, Yong
Barry Menglong Yao, Aditya Shah, Lichao Sun, Jin-Hee Cheng, Xavier Garcia, Jonathan Shen, and Orhan
Cho, and Lifu Huang. 2022. End-to-end multimodal Firat. 2022. Examining scaling and transfer of lan-
fact-checking and explanation generation: A chal- guage model architectures for machine translation.
lenging dataset and models. Proceedings of the 46th In Proceedings of the 39th International Conference
International ACM SIGIR Conference on Research on Machine Learning, volume 162 of Proceedings
and Development in Information Retrieval. of Machine Learning Research, pages 26176–26192.
PMLR.
Rui Ye, Wenhao Wang, Jingyi Chai, Dihan Li, Zexi
Li, Yinda Xu, Yaxin Du, Yanfeng Wang, and Siheng Jingyi Zhang, Jiaxing Huang, Sheng Jin, and Shijian Lu.
Chen. 2024. Openfedllm: Training large language 2024a. Vision-language models for vision tasks: A
models on decentralized private data via federated survey. IEEE Transactions on Pattern Analysis and
learning. ArXiv, abs/2402.06954. Machine Intelligence, pages 1–20.

Kaiyan Zhang, Jianyu Wang, Ermo Hua, Biqing Qi,

Le Yu, Bowen Yu, Haiyang Yu, Fei Huang, and Yongbin Ning Ding, and Bowen Zhou. 2024b. Cogenesis: A
Li. 2023. Language models are super mario: Absorb- framework collaborating large and small language
ing abilities from homologous models as a free lunch. models for secure context-aware instruction follow-
CoRR, abs/2311.03099. ing. ArXiv, abs/2403.03129.
Yao-Ching Yu, Chun-Chih Kuo, Ziqi Ye, Yu-Cheng Muru Zhang, Ofir Press, William Merrill, Alisa
Chang, and Yueh-Se Li. 2024. Breaking the ceiling Liu, and Noah A Smith. 2023a. How language
of the llm community by treating token generation model hallucinations can snowball. arXiv preprint
as a classification for ensembling. arXiv preprint arXiv:2305.13534.
arXiv:2406.12585.
Shun Zhang, Zhenfang Chen, Sunli Chen, Yikang Shen,
Hongyi Yuan, Keming Lu, Fei Huang, Zheng Yuan, and Zhiqing Sun, and Chuang Gan. 2024c. Improving
Chang Zhou. 2023. Speculative contrastive decoding. reinforcement learning from human feedback with
arXiv preprint arXiv:2311.08981. efficient reward model ensemble. arXiv preprint
arXiv:2401.16635.
Weizhe Yuan, Graham Neubig, and Pengfei Liu. 2021.
Bartscore: Evaluating generated text as text gener- Tianyi Zhang*, Varsha Kishore*, Felix Wu*, Kilian Q.
ation. Advances in Neural Information Processing Weinberger, and Yoav Artzi. 2020. Bertscore: Eval-
Systems, 34:27263–27277. uating text generation with bert. In International
Conference on Learning Representations.
Murong Yue, Jie Zhao, Min Zhang, Liang Du, and Ziyu
Yao. 2024. Large language model cascades with Yue Zhang, Leyang Cui, Wei Bi, and Shuming Shi.
mixture of thought representations for cost-efficient 2023b. Alleviating hallucinations of large lan-
reasoning. In The Twelfth International Conference guage models through induced hallucinations. arXiv
on Learning Representations. preprint arXiv:2312.15710.

28
Haiyan Zhao, Hanjie Chen, Fan Yang, Ninghao Liu,
Huiqi Deng, Hengyi Cai, Shuaiqiang Wang, Dawei
Yin, and Mengnan Du. 2024a. Explainability for
large language models: A survey. ACM Trans. Intell.
Syst. Technol., 15(2).
Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang,
Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen
Zhang, Junjie Zhang, Zican Dong, et al. 2023. A
survey of large language models. arXiv preprint
arXiv:2303.18223.
Xuandong Zhao, Xianjun Yang, Tianyu Pang, Chao Du,
Lei Li, Yu-Xiang Wang, and William Yang Wang.
2024b. Weak-to-strong jailbreaking on large lan-
guage models. arXiv preprint arXiv:2401.17256.
Yuyan Zhou, Liang Song, Bingning Wang, and Weipeng
Chen. 2024a. Metagpt: Merging large language mod-
els using model exclusive task arithmetic. Preprint,
arXiv:2406.11385.
Zhanhui Zhou, Zhixuan Liu, Jie Liu, Zhichen Dong,
Chao Yang, and Yu Qiao. 2024b. Weak-to-strong
search: Align large language models via search-
ing over small language models. arXiv preprint
arXiv:2405.19262.
Zixuan Zhou, Xuefei Ning, Ke Hong, Tianyu Fu, Ji-
aming Xu, Shiyao Li, Yuming Lou, Luning Wang,
Zhihang Yuan, Xiuhong Li, Shengen Yan, Guohao
Dai, Xiao-Ping Zhang, Yuhan Dong, and Yu Wang.
2024c. A survey on efficient inference for large lan-
guage models. ArXiv, abs/2404.14294.

Yaochen Zhu, Rui Xia, and Jiajun Zhang. 2024. DPPA:

pruning method for large language model to model
merging. CoRR, abs/2403.02799.

29