Prompting
Sunday, October 20, 2024 5:01 PM
1. Learning from Demonstration: Few-shot Prompting
- A prompt is a text string that a user issues to a language model to get the model to do something
useful. Thus, the prompt creates a context that guides LLMs to generate useful outputs to achieve
some user goal.
○ The process of finding effective prompts for a task is known as prompt engineering.
- With suitable additions to the context a single LLM can produce outputs appropriate for many
different tasks.
E.g.1:
○ A summary
○ Whether the review was truthful or likely to have been fabricated
○ A translation to another language
- Each template consists of an input text, designed as {input}, followed by a verbatim prompt to be
passed to an LLM. These templates are applied to inputs to create filled prompts - instantiated
prompts suitable for use as inputs to an LLM.
- Consider the following example:
Translate English to French:
Did not like the service that I was provided!
This prompt doesn't do a good job of constraining possible continuations. Instead of a French
translation, models given this prompt may instead generate another sentence in English that
simply extends the English review.
=> Prompts need to be designed unambiguously, so that any reasonable continuation would
accomplish the desired task.
E.g.2:
Model Alignment, Prompting and In-Context Learning Page 1
- We can also prompt the system to break down complex task, using methods like chain-of-
thought. Or we may want to restrict a summary to be a particular length, to have an answer
generated according to some kind of persona or role.
- In summary, we prompt an LM by transforming each task into a form that is amenable to
contextual generation by an LLM, as follows:
1. For a given task, develop a task-specific template that has a free parameter for the input
text
2. Given that input and the task-specific template, the input is used to instantiate a filled
prompt that is then passed to a pretrained language model
3. Autoregressive decoding is then used to generate a sequence of token outputs.
4. The output of the model can either be used directly as the desired output (as in the case of
naturally generative tasks such as translation or summarization), or a task-appropriate
answer can be extracted from the generated output (as in the case of classification).
A. Learning from Demonstrations: Few-Shot Prompting
- We can improve a prompt by including some labeled examples in the prompt template:
demonstrations
○ The task of prompting with examples is sometimes called few-shot prompting, as
contrasted with zero-shot prompting which means instructions that don't include labeled
examples.
Model Alignment, Prompting and In-Context Learning Page 2
- How many demonstrations?
The number of demonstrations doesn’t have to be large. A small number of randomly selected
labeled examples used as demonstrations can be sufficient to improve performance over the zero -
shot setting. Indeed, the largest performance gains in few-shot prompting tends to come from the
first training example, with diminishing returns for subsequent demonstrations
- Why isn't it useful to have more demonstrations?
The reason is the primary benefit in examples is to demonstrate the task to be performed by an
LLM and the format of the sequence, not to provide relevant information as to the right answer
for any particular question.
In fact, demonstrations that have incorrect answers can still improve a system. Adding too many
examples seems to cause the model to overfit to details of the exact examples chosen and
generalize poorly.
- How to Select Demonstrations?
Generally, the best way to select demonstrations from the training set is programmatically:
choosing the set of demonstrations that most increases task performance of the prompt on a test
set. Task performance for sentiment analysis or multiple-choice question answering can be
measured in accuracy; for machine translation with chrF, and for summarization via Rouge .
B. In-Context Learning and Induction Heads
- Learning via pretraining means updating the model's parameters by using gradient descent over
some loss function >< Prompting with demonstrations can teach a model to do a new task.
- We use the term in-context learning to refer to the kinds of learning that language model do from
their prompts. In-context learning means language models learning to do new tasks, better predict
tokens, or generally reduce their loss, but without any gradient -based updates to the model’s
parameters.
- How does in-context learning work?
○ One hypothesis is based on the idea of induction heads. The induction head circuit is part of
the attention computation in transformers, discovered by looking at mini language models
with only 1-2 attention heads.
○ The function of the induction head is to predict repeated sequences. For example, if it sees
the pattern AB...A in an input sequence, it predicts that B will follow, instantiating the
Model Alignment, Prompting and In-Context Learning Page 3
the pattern AB...A in an input sequence, it predicts that B will follow, instantiating the
pattern completion rule AB...A! B.
It does this by having a prefix matching component of the attention computation that, when
looking at the current token A, searches back over the context to find a prior instance of A. If
it finds one, the induction head has a copying mechanism that “copies” the token B that
followed the earlier A, by increasing the probability the B will occur next.
• Further Readings:
Model Alignment, Prompting and In-Context Learning Page 4
Post-training and Model Alignment
Saturday, October 26, 2024 10:07 PM
- There are limits to how much can be expected from a model whose sole training objective is to
predict the next word from large amounts of pretraining text.
- To see this, consider the following example:
○ Here, the LLM ignores the intent of the request and relies instead on its natural inclination
to autoregressively generate continuations consistent with its context.
○ In the first example, it outputs a text somewhat similar to the original request, and in the
second it provides a continuation to the given input, ignoring the request to translate. LLMs
are not sufficiently helpful: they need extra training to increase their abilities to follow
textual instructions.
- A deeper problem is that LLMs can simultaneously be too harmful. For example, they can
generate text that is false, including unsafe misinformation, or text that is toxic in many ways,
such as facilitating the spread of hate speech. Even completely non-toxic prompts can lead LLMs
to output hate speech and abuse their users.
=> In attempt to address these two problems, LMs generally include two additional kinds of training
for model alignment: methods designed to adjust LLMs to better align them to human needs for models
to be helpful.
○ Instruction Tuning (SFT for supervised finetuning): models are finetuned on a corpus of
instructions and questions with their corresponding responses.
○ Preference Alignment: Reinforcement Learning from Human Feedback (RLHF)
- We'll use the term base model to mean a model that has been pretrained but hasn't yet been
aligned either by instruction tuning or RLHF. And we'll refer to these steps as post-training,
meaning that they apply after the model has been pretrained.
Model Alignment, Prompting and In-Context Learning Page 5
Model Alignment: Instruction Tuning
Sunday, October 27, 2024 9:38 AM
- Instruction Tuning: is a method for making an LLM better at following instructions. It involves
taking a base pretrained LLM and training it to follow instructions for a range of tasks, from
Machine Translation to Meal Planning.
○ The resulting model engages in a form of meta-learning - it improves its ability to follow
instructions generally.
- Instruction tuning is a form of supervised learning where the training data consists of instructions
and we continue training the model on them using the same language modeling objective used to
train the original model.
- Each instruction or question in the instruction tuning data has a supervised objective: a correct
answer to the question or a response to the instruction.
A. Instructions as Training Data
- The instruction-tuning datasets are created in 4 ways:
○ Write the instances directly. E.g: Aya instruct finetuning corpus
○ Make use of the copious amounts of supervised training data that have been curated over
the years for a wide range of natural language tasks. E.g: SQuAD datasets. These data can be
automatically converted into sets of instruction prompts and input/output demonstration
pairs via simple templates
Model Alignment, Prompting and In-Context Learning Page 6
pairs via simple templates
○ Because supervised NLP datasets are themselves often produced by crowdworkers based on
carefully written annotation guidelines, we can draw on these guidelines, which can include
detailed step-by-step instructions, pitfalls to avoid, formatting instructions, etc. These
annotation guidelines can be used directly as prompts to a language model to create
instruction-tuning training examples
Model Alignment, Prompting and In-Context Learning Page 7
○ Use language models to help at each stage. For example:
B. Evaluation of Instruction-Tuned Models
- In assessing instruction-tuning methods, we need to assess how well an instruction-trained model
platforms on novel tasks for which it has not been given explicit instructions.
- Leave-one-out approach: instruction-tune a model on some large set of tasks and then assess it
on a withheld task.
○ Problem: The enormous numbers of tasks in instruction-tuning datasets often overlap.
- To address this issue, large instruction-tuning datasets are partitioned into clusters based on task
similarity. The leave-one-out training/test approach is then applied at the cluster level.
For example:
Model Alignment, Prompting and In-Context Learning Page 8
For example:
Model Alignment, Prompting and In-Context Learning Page 9
Chain-of-Thought Prompting
Sunday, October 27, 2024 10:34 AM
- Goal: improve performance on difficult reasoning tasks that language models tend to fail on.
- Each of the demonstrations in the few-shot prompt is augmented with some text explaining some
reasoning steps. We want the model to output similar kinds of reasoning steps for the problem
being solved, and for the output of those reasoning steps to cause the system to generate the
correct answer.
Model Alignment, Prompting and In-Context Learning Page 10
Automatic Prompt Optimization
Sunday, October 27, 2024 10:38 AM
- Given a prompt for a task (human or computer generated), prompt optimization methods search
for prompts with improved performance. Most of these approaches can be viewed as a form of
iterative improvement search through a space of possible prompts for those that optimize
performance on a task.
- These approaches all share the following components:
○ A start state - An initial human or machine generated prompt or prompts suitable for some
task
○ A scoring metric - A method for assessing how well a given prompt performs on the task
○ An explanation method - A method for generating variations of a prompt.
- Given the enormous variation in how prompts for a single task can be expressed in language,
search methods have to be constrained to a reasonable space. Beam-search is widely used
method that combines BFS with a fixed-width priority queue that focuses the search effort on the
top performing variants.
A. Candidate Scoring
- Candidate scoring methods assess the likely performance of potential prompts, both to identify
promising avenues of search and to prune those that are unlikely to be effective. Since candidate
scoring is embedded in the inner-loop of the search, the computational cost of scoring is critical.
- Given access to labeled training data, candidate prompts can be scored based on execution
accuracy. In this approach, candidate prompts are combined with inputs sampled from the
training data and passed to an LLM for decoding. The LLM output is evaluated against the training
Model Alignment, Prompting and In-Context Learning Page 11
training data and passed to an LLM for decoding. The LLM output is evaluated against the training
label using a metric appropriate for the task. In the case of classification-based tasks, this is
effectively a 0/1 loss — how many examples were correctly labeled with the given prompt.
Generative applications such as summarization or translation use task-specific similarity scores
such as BERTScore, Bleu, or ROUGE
- Given the computational cost of issuing calls to an LLM, evaluating each candidate prompt against
a complete training set would be infeasible. Instead, prompt performance is estimated from a
small sample of training data.
B. Prompt Expansion
- Prompt expansion generates variants of a given prompt to create an expanded set of neighboring
prompts that may improve performance over the original.
- A common method is to use the language model to create paraphrases:
- A variation of this method is to truncate the current prompt at set of random locations,
generating a set of prompt prefixes. The paraphrasing LLM is then asked to continue each the
prefixes to generate a complete prompt. => uniformed search
- By contrast, Prasad employ a candidate expansion technique that explicitly attempts to generate
superior prompts during the expansion process:
In this approach, the current candidate is first applied to a sample of training examples using the
execution accuracy approach. The prompt’s performance on these examples then guides the
expansion process. Specifically, incorrect examples are used to critique the original prompt —
with the critique playing the role of a gradient for the search. The method includes the following
steps:
1. Run the prompt on a sample of training examples
2. Identify examples where the prompt fails
3. Ask an LLM to produce to critique of the prompt in light of the failed examples
4. Provide the resulting critique to an LLM, and ask it to generate improved prompts.
Model Alignment, Prompting and In-Context Learning Page 12
Model Alignment, Prompting and In-Context Learning Page 13
Evaluating Prompted Language Models
Sunday, October 27, 2024 11:26 AM
Model Alignment, Prompting and In-Context Learning Page 14
Model Alignment, Prompting and In-Context Learning Page 15
Model Alignment with Human Preferences: RLHF and DPO
Sunday, October 27, 2024 11:28 AM
Model Alignment, Prompting and In-Context Learning Page 16