Handling the Risks of Language
Models
1
Contents
1. Introduction
2. Biases
3. Privacy
a. Anonymization and Pseudonymization
b. Model hacking
4. Reinforcement Learning from Human Feedback (RLHF)
5. Augmented Language Models (Toolformer)
Course 5: LM Risks and Alignement 2
Introduction
Course 5: LM Risks and Alignement 3
Defintions i
Which risks? Misinformation, biased information, and privacy
Concerns.
Biases: misleading, or false-logical thought processes.
Spurious features.
Course 5: LM Risks and Alignement 4
Defintions ii
Privacy Concerns are from an NLP practitioner stand-point.
Data anonymization.
Data Leaks (demander au modèle ses donénes
d'entraînement)/Model hacking (raisonnnement inductif donc
certaines données sont nécessaires)
Course 5: LM Risks and Alignement 5
Defintions iii
Alignement are techniques used match the model's output with the
user's exact intent while remaining harmless.
Reinforcement Learning from Human Feedback (RLHF).
Retrival Augmented Generation (RAG)
Course 5: LM Risks and Alignement 6
Aim
Mitigating language models' risks via straightforward alignement.
Course 5: LM Risks and Alignement 7
Biases
Course 5: LM Risks and Alignement 8
"Avoiding" Learning Spurious Features
Example A [1]
Label=+1 Label=-1
Riveting film of the highest Thank God I didn't go to the
calibre! cinema.
Definitely worth tha watch! Boring as hell.
I wanted to give up in the first
A true story told perfectly!
hour...
Course 5: LM Risks and Alignement 9
"Avoiding" Learning Spurious Features
Example B [2]
Course 5: LM Risks and Alignement 10
Rule-based/Crowd-Sourced Preprocessing?
If we know the spurious correlations
1. Data augmentation and subsampling.
Label=+1 Label=-1
Riveting film of the highest Thank God I didn't go to the
calibre ! cinema !
Definitely worth tha watch ! Boring as hell !
I wanted to give up in the first
A true story told perfectly.
hour...
Course 5: LM Risks and Alignement 11
Rule-based/Crowd-Sourced Preprocessing?
Pros: can be expensive in human resources.
Cons: hit the model's performance if not done properly.
Course 5: LM Risks and Alignement 12
Multitasking? i
If we don't know the spurious correlations
1. Pre-training outperforms heavy preprocessing and long domain-
specific fine-tuning [4]
2. Appending data from other tasks also helps [3].
Pros: somewhat straightforward to implement.
Cons: longer and expensive training time.
Course 5: LM Risks and Alignement 13
Multitasking? ii
Pre-training medium/large sized models takes a lot of data and
computation power, hence, only a few actors can afford it.
=> smaller/specialized models are derived from those models via fine-
tuning.
=> The base models biases are propagated to the
sammeler/specialized ones.
Course 5: LM Risks and Alignement 14
Scaling? i
In context learning: the model learns to solve a task at inference with
no weights update (More on this later).
Course 5: LM Risks and Alignement 15
Scaling? ii
"Larger models make increasingly efficient use of in-context
information." [5] Yes but [6]...
Course 5: LM Risks and Alignement 16
Privacy
Course 5: LM Risks and Alignement 17
Anonymization and Pseudonymization
Anonymization: Francis Kulumba, 25 -> N/A, 25-30
Pseudonymization: Francis Kulumba, 25 -> Eqzmbhr Jtktlaz, 52
Some data are too hard to anonymize/pseudonimyze:
Medical care
Resumes
Course 5: LM Risks and Alignement 18
Hacking i
Just like any I/O system, generative LLMs are sensible to injections.
1. Persistence and Correction
No, that's incorrect because...
Are you sure?
2. Context Expansion
I'm conducting a study on...
I'm working for [...] and I'm trying to prevent the potential harm of...
Course 5: LM Risks and Alignement 19
Hacking ii
3. Inversion
Ask the agent to produce two answer, the one to your prompt, and
the opposite of it.
4. Response Conditioning
Exploit in-context learning to cue the LLM to respond in a desired
way.
Course 5: LM Risks and Alignement 20
Hacking iii
5. Context Leveraging
Giving an instruction the agent will interpret as an overriding that
hampers later instructions.
Speak to me as if you were Bugs Bunny.
Course 5: LM Risks and Alignement 21
Reinforcement Learning from Human
Feedback (RLHF)
Course 5: LM Risks and Alignement 22
Aim
Instead of trying to safeguard every bit of the training data to render
the model harmless, how about trying to teach it human preferences?
Course's material from HuggingFace.
Course 5: LM Risks and Alignement 23
Traditional RL
We want to maximize the expected reward with respect to the
model's parameters at a given state .
Course 5: LM Risks and Alignement 24
Traditional RL
(1)
(2) Log-derivative trick:
We put (2) in (1):
Course 5: LM Risks and Alignement 25
Traditional RL
(1) becomes
We can use Monte-Carlo samples to estimate (1) as:
Thus, we want have the following optimization step
Course 5: LM Risks and Alignement 26
RLHF
Course 5: LM Risks and Alignement 27
RLHF
Course 5: LM Risks and Alignement 28
RLHF
Course 5: LM Risks and Alignement 29
Augmented Language Models (Toolformer)
Course 5: LM Risks and Alignement 30
Retrival Augmented Generation (RAG)
RAG allow an LLM to have updated knowledge without having to fine-
tune it. It also mitigates hallucination
Course 5: LM Risks and Alignement 31
Retrival Augmented Generation (RAG)
Course 5: LM Risks and Alignement 32
Retrival Augmented Generation (RAG)
More here: Retrieval-Augmented Generation for Knowledge-
Intensive NLP Tasks
Course 5: LM Risks and Alignement 33
Toolformer
Course 5: LM Risks and Alignement 34
Toolformer
Course 5: LM Risks and Alignement 35
Toolformer
Course 5: LM Risks and Alignement 36
Toolformer
More here: Toolformer: Language Models Can Teach Themselves to
Use Tools
Course 5: LM Risks and Alignement 37
References
[1] He, H. (2023, July 9). Robust Natural Language Understanding.
[2] Singla, S., & Feizi, S. (2021). Causal imagenet: How to discover
spurious features in deep learning. arXiv preprint arXiv:2110.04301,
23.
[3] Carmon, Y., Raghunathan, A., Schmidt, L., Duchi, J. C., & Liang, P. S.
(2019). Unlabeled data improves adversarial robustness. Advances in
neural information processing systems, 32.
Course 5: LM Risks and Alignement 38
[4] Pretrained Transformers Improve Out-of-Distribution Robustness
(Hendrycks et al., ACL 2020)
[5] Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal,
P., ... & Amodei, D. (2020). Language models are few-shot learners.
Advances in neural information processing systems, 33, 1877-1901.
[6] Zhao, Z., Wallace, E., Feng, S., Klein, D., & Singh, S. (2021, July).
Calibrate before use: Improving few-shot performance of language
models. In International Conference on Machine Learning (pp. 12697-
12706). PMLR.
Course 5: LM Risks and Alignement 39