This model was contributed to Hugging Face Transformers on 2020-11-16.

GPT-2

GPT-2 is a scaled up version of GPT, a causal transformer language model, with 10x more parameters and training data. The model was pretrained on a 40GB dataset to predict the next word in a sequence based on all the previous words. This approach enabled the model to perform many downstream tasks in a zero-shot setting. The blog post released by OpenAI can be found here.

The model architecture uses a unidirectional (causal) attention mechanism where each token can only attend to previous tokens, making it particularly effective for text generation tasks.

You can find all the original GPT-2 checkpoints under the OpenAI community organization.

Click on the GPT-2 models in the right sidebar for more examples of how to apply GPT-2 to different language tasks.

The example below demonstrates how to generate text with Pipeline or the AutoModel, and from the command line.

Pipeline

AutoModel

One can also serve the model using vLLM with the transformers backend.

vllm serve openai-community/gpt2 --model-imp transformers

Quantization reduces the memory burden of large models by representing the weights in a lower precision. Refer to the Quantization overview for more available quantization backends.

The example below uses bitsandbytes to only quantize the weights to 4-bits.

from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig


quantization_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype="float16",
    bnb_4bit_use_double_quant=True
)

model = AutoModelForCausalLM.from_pretrained(
    "openai-community/gpt2-xl",
    quantization_config=quantization_config,
    device_map="auto"
)

tokenizer = AutoTokenizer.from_pretrained("openai-community/gpt2-xl")
inputs = tokenizer("Once upon a time, there was a magical forest", return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=100)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Notes

Pad inputs on the right because GPT-2 uses absolute position embeddings.
GPT-2 can reuse previously computed key-value attention pairs. Access this feature with the past_key_values parameter in GPT2Model.forward().
Enable the scale_attn_by_inverse_layer_idx and reorder_and_upcast_attn parameters to apply the training stability improvements from Mistral.

GPT2Config

class transformers.GPT2Config

< source >

( transformers_version: str | None = Nonearchitectures: list[str] | None = Noneoutput_hidden_states: bool | None = Falsereturn_dict: bool | None = Truedtype: typing.Union[str, ForwardRef('torch.dtype'), NoneType] = Nonechunk_size_feed_forward: int = 0is_encoder_decoder: bool = Falseid2label: dict[int, str] | dict[str, str] | None = Nonelabel2id: dict[str, int] | dict[str, str] | None = Noneproblem_type: typing.Optional[typing.Literal['regression', 'single_label_classification', 'multi_label_classification']] = Nonevocab_size: int = 50257n_positions: int = 1024n_embd: int = 768n_layer: int = 12n_head: int = 12n_inner: int | None = Noneactivation_function: str = 'gelu_new'resid_pdrop: float | int = 0.1embd_pdrop: float | int = 0.1attn_pdrop: float | int = 0.1layer_norm_epsilon: float = 1e-05initializer_range: float = 0.02summary_type: str = 'cls_index'summary_use_proj: bool = Truesummary_activation: str | None = Nonesummary_proj_to_labels: bool = Truesummary_first_dropout: float | int = 0.1scale_attn_weights: bool = Trueuse_cache: bool = Truebos_token_id: int | None = 50256eos_token_id: int | list[int] | None = 50256pad_token_id: int | None = Nonescale_attn_by_inverse_layer_idx: bool = Falsereorder_and_upcast_attn: bool = Falseadd_cross_attention: bool = Falsetie_word_embeddings: bool = True )

Parameters

vocab_size (int, optional, defaults to 50257) — Vocabulary size of the model. Defines the number of different tokens that can be represented by the input_ids.
n_positions (int, optional, defaults to 1024) — The maximum sequence length that this model might ever be used with.
n_embd (int, optional, defaults to 768) — Dimensionality of the embeddings and hidden states.
n_layer (int, optional, defaults to 12) — Number of hidden layers in the Transformer decoder.
n_head (int, optional, defaults to 12) — Number of attention heads for each attention layer in the Transformer decoder.
n_inner (int, optional) — Dimension of the MLP representations.
activation_function (str, optional, defaults to gelu_new) — The non-linear activation function (function or string) in the decoder. For example, "gelu", "relu", "silu", etc.
resid_pdrop (Union[float, int], optional, defaults to 0.1) — The dropout probability for all fully connected layers in the embeddings, encoder, and pooler.
embd_pdrop (Union[float, int], optional, defaults to 0.1) — The dropout ratio for the embeddings.
attn_pdrop (Union[float, int], optional, defaults to 0.1) — The dropout ratio for the attention probabilities.
layer_norm_epsilon (float, optional, defaults to 1e-05) — The epsilon used by the layer normalization layers.
initializer_range (float, optional, defaults to 0.02) — The standard deviation of the truncated_normal_initializer for initializing all weight matrices.
summary_type (string, optional, defaults to "cls_index") — Argument used when doing sequence summary, used in the models GPT2DoubleHeadsModel. Has to be one of the following options:
- "last": Take the last token hidden state (like XLNet).
- "first": Take the first token hidden state (like BERT).
- "mean": Take the mean of all tokens hidden states.
- "cls_index": Supply a Tensor of classification token position (like GPT/GPT-2).
- "attn": Not implemented now, use multi-head attention.
summary_use_proj (bool, optional, defaults to True) — Argument used when doing sequence summary, used in the models GPT2DoubleHeadsModel. Whether or not to add a projection after the vector extraction.
summary_activation (str, optional) — Argument used when doing sequence summary. Used in for the multiple choice head in GPT2DoubleHeadsModel. Pass "tanh" for a tanh activation to the output, any other value will result in no activation.
summary_proj_to_labels (bool, optional, defaults to True) — Argument used when doing sequence summary, used in the models GPT2DoubleHeadsModel. Whether the projection outputs should have config.num_labels or config.hidden_size classes.
summary_first_dropout (float, optional, defaults to 0.1) — Argument used when doing sequence summary, used in the models GPT2DoubleHeadsModel. The dropout ratio to be used after the projection and activation.
scale_attn_weights (bool, optional, defaults to True) — Whether to scale embeddings by dividing by sqrt(d_model).
use_cache (bool, optional, defaults to True) — Whether or not the model should return the last key/values attentions (not used by all models). Only relevant if config.is_decoder=True or when the model is a decoder-only generative model.
bos_token_id (int, optional, defaults to 50256) — Token id used for beginning-of-stream in the vocabulary.
eos_token_id (Union[int, list[int]], optional, defaults to 50256) — Token id used for end-of-stream in the vocabulary.
pad_token_id (int, optional) — Token id used for padding in the vocabulary.
scale_attn_by_inverse_layer_idx (bool, optional, defaults to False) — Whether to additionally scale attention weights by 1 / layer_idx + 1.
reorder_and_upcast_attn (bool, optional, defaults to False) — Whether to scale keys (K) prior to computing attention (dot-product) and upcast attention dot-product/softmax to float() when training with mixed precision.
add_cross_attention (bool, optional, defaults to False) — Whether cross-attention layers should be added to the model.
tie_word_embeddings (bool, optional, defaults to True) — Whether to tie weight embeddings according to model’s tied_weights_keys mapping.

This is the configuration class to store the configuration of a GPT2Model. It is used to instantiate a Gpt2 model according to the specified arguments, defining the model architecture. Instantiating a configuration with the defaults will yield a similar configuration to that of the openai-community/gpt2

Configuration objects inherit from PreTrainedConfig and can be used to control the model outputs. Read the documentation from PreTrainedConfig for more information.

Example:

>>> from transformers import GPT2Config, GPT2Model

>>> # Initializing a GPT2 configuration
>>> configuration = GPT2Config()

>>> # Initializing a model (with random weights) from the configuration
>>> model = GPT2Model(configuration)

>>> # Accessing the model configuration
>>> configuration = model.config

GPT2Tokenizer

class transformers.GPT2Tokenizer

< source >

Parameters

vocab_file (str) — Path to the vocabulary file.
merges_file (str) — Path to the merges file.
errors (str, optional, defaults to "replace") — Paradigm to follow when decoding bytes to UTF-8. See bytes.decode for more information.
unk_token (str, optional, defaults to "<|endoftext|>") — The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this token instead.
bos_token (str, optional, defaults to "<|endoftext|>") — The beginning of sequence token.
eos_token (str, optional, defaults to "<|endoftext|>") — The end of sequence token.
pad_token (str, optional) — The token used for padding, for example when batching sequences of different lengths.
add_prefix_space (bool, optional, defaults to False) — Whether or not to add an initial space to the input. This allows to treat the leading word just as any other word. (GPT2 tokenizer detect beginning of words by the preceding space).
add_bos_token (bool, optional, defaults to False) — Whether or not to add an initial beginning of sentence token to the input. This allows to treat the leading word just as any other word.
vocab (str or dict[str, int], optional) — Custom vocabulary dictionary. If not provided, vocabulary is loaded from vocab_file.
merges (str or list[str], optional) — Custom merges list. If not provided, merges are loaded from merges_file.

Construct a GPT-2 tokenizer. Based on byte-level Byte-Pair-Encoding.

This tokenizer has been trained to treat spaces like parts of the tokens (a bit like sentencepiece) so a word will

be encoded differently whether it is at the beginning of the sentence (without space) or not:

>>> from transformers import GPT2Tokenizer

>>> tokenizer = GPT2Tokenizer.from_pretrained("openai-community/gpt2")
>>> tokenizer("Hello world")["input_ids"]
[15496, 995]

>>> tokenizer(" Hello world")["input_ids"]
[18435, 995]

You can get around that behavior by passing add_prefix_space=True when instantiating this tokenizer or when you call it on some text, but since the model was not pretrained this way, it might yield a decrease in performance.

When used with is_split_into_words=True, this tokenizer will add a space before each word (even the first one).

This tokenizer inherits from TokenizersBackend which contains most of the main methods. Users should refer to this superclass for more information regarding those methods.

save_vocabulary

< source >

( save_directory: strfilename_prefix: str | None = None )

GPT2TokenizerFast

class transformers.GPT2Tokenizer

< source >

Parameters

vocab_file (str) — Path to the vocabulary file.
merges_file (str) — Path to the merges file.
errors (str, optional, defaults to "replace") — Paradigm to follow when decoding bytes to UTF-8. See bytes.decode for more information.
unk_token (str, optional, defaults to "<|endoftext|>") — The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this token instead.
bos_token (str, optional, defaults to "<|endoftext|>") — The beginning of sequence token.
eos_token (str, optional, defaults to "<|endoftext|>") — The end of sequence token.
pad_token (str, optional) — The token used for padding, for example when batching sequences of different lengths.
add_prefix_space (bool, optional, defaults to False) — Whether or not to add an initial space to the input. This allows to treat the leading word just as any other word. (GPT2 tokenizer detect beginning of words by the preceding space).
add_bos_token (bool, optional, defaults to False) — Whether or not to add an initial beginning of sentence token to the input. This allows to treat the leading word just as any other word.
vocab (str or dict[str, int], optional) — Custom vocabulary dictionary. If not provided, vocabulary is loaded from vocab_file.
merges (str or list[str], optional) — Custom merges list. If not provided, merges are loaded from merges_file.

Construct a GPT-2 tokenizer. Based on byte-level Byte-Pair-Encoding.

This tokenizer has been trained to treat spaces like parts of the tokens (a bit like sentencepiece) so a word will

be encoded differently whether it is at the beginning of the sentence (without space) or not:

>>> from transformers import GPT2Tokenizer

>>> tokenizer = GPT2Tokenizer.from_pretrained("openai-community/gpt2")
>>> tokenizer("Hello world")["input_ids"]
[15496, 995]

>>> tokenizer(" Hello world")["input_ids"]
[18435, 995]

When used with is_split_into_words=True, this tokenizer will add a space before each word (even the first one).

This tokenizer inherits from TokenizersBackend which contains most of the main methods. Users should refer to this superclass for more information regarding those methods.

GPT2 specific outputs

Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads.

Base class for outputs of models predicting if two sentences are consecutive or not.

GPT2Model

If past_key_values is used, only input_ids that do not have their past calculated should be passed as input_ids.

Only Cache instance is allowed as input, see our kv cache guide. If no past_key_values are passed, DynamicCache will be initialized by default.

A BaseModelOutputWithPastAndCrossAttentions or a tuple of torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various elements depending on the configuration (GPT2Config) and inputs.

GPT2LMHeadModel

If past_key_values is used, only input_ids that do not have their past calculated should be passed as input_ids.

Only Cache instance is allowed as input, see our kv cache guide. If no past_key_values are passed, DynamicCache will be initialized by default.

A CausalLMOutputWithCrossAttentions or a tuple of torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various elements depending on the configuration (GPT2Config) and inputs.

Example:

>>> import torch
>>> from transformers import AutoTokenizer, GPT2LMHeadModel

>>> tokenizer = AutoTokenizer.from_pretrained("openai-community/gpt2")
>>> model = GPT2LMHeadModel.from_pretrained("openai-community/gpt2")

>>> inputs = tokenizer("Hello, my dog is cute", return_tensors="pt")
>>> outputs = model(**inputs, labels=inputs["input_ids"])
>>> loss = outputs.loss
>>> logits = outputs.logits

GPT2DoubleHeadsModel

If past_key_values is used, only input_ids that do not have their past calculated should be passed as input_ids.

Only Cache instance is allowed as input, see our kv cache guide. If no past_key_values are passed, DynamicCache will be initialized by default.

A GPT2DoubleHeadsModelOutput or a tuple of torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various elements depending on the configuration (GPT2Config) and inputs.

The GPT2DoubleHeadsModel forward method, overrides the __call__ special method.

Example:

>>> import torch
>>> from transformers import AutoTokenizer, GPT2DoubleHeadsModel

>>> tokenizer = AutoTokenizer.from_pretrained("openai-community/gpt2")
>>> model = GPT2DoubleHeadsModel.from_pretrained("openai-community/gpt2")

>>> # Add a [CLS] to the vocabulary (we should train it also!)
>>> num_added_tokens = tokenizer.add_special_tokens({"cls_token": "[CLS]"})
>>> # Update the model embeddings with the new vocabulary size
>>> embedding_layer = model.resize_token_embeddings(len(tokenizer))

>>> choices = ["Hello, my dog is cute [CLS]", "Hello, my cat is cute [CLS]"]
>>> encoded_choices = [tokenizer.encode(s) for s in choices]
>>> cls_token_location = [tokens.index(tokenizer.cls_token_id) for tokens in encoded_choices]

>>> input_ids = torch.tensor(encoded_choices).unsqueeze(0)  # Batch size: 1, number of choices: 2
>>> mc_token_ids = torch.tensor([cls_token_location])  # Batch size: 1

>>> outputs = model(input_ids, mc_token_ids=mc_token_ids)
>>> lm_logits = outputs.logits
>>> mc_logits = outputs.mc_logits

GPT2ForQuestionAnswering

If past_key_values is used, only input_ids that do not have their past calculated should be passed as input_ids.

A QuestionAnsweringModelOutput or a tuple of torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various elements depending on the configuration (GPT2Config) and inputs.

Example:

>>> from transformers import AutoTokenizer, GPT2ForQuestionAnswering
>>> import torch

>>> tokenizer = AutoTokenizer.from_pretrained("openai-community/gpt2")
>>> model = GPT2ForQuestionAnswering.from_pretrained("openai-community/gpt2")

>>> question, text = "Who was Jim Henson?", "Jim Henson was a nice puppet"

>>> inputs = tokenizer(question, text, return_tensors="pt")
>>> with torch.no_grad():
...     outputs = model(**inputs)

>>> answer_start_index = outputs.start_logits.argmax()
>>> answer_end_index = outputs.end_logits.argmax()

>>> predict_answer_tokens = inputs.input_ids[0, answer_start_index : answer_end_index + 1]
>>> tokenizer.decode(predict_answer_tokens, skip_special_tokens=True)
...

>>> # target is "nice puppet"
>>> target_start_index = torch.tensor([14])
>>> target_end_index = torch.tensor([15])

>>> outputs = model(**inputs, start_positions=target_start_index, end_positions=target_end_index)
>>> loss = outputs.loss
>>> round(loss.item(), 2)
...

GPT2ForSequenceClassification

If past_key_values is used, only input_ids that do not have their past calculated should be passed as input_ids.

Only Cache instance is allowed as input, see our kv cache guide. If no past_key_values are passed, DynamicCache will be initialized by default.

A SequenceClassifierOutputWithPast or a tuple of torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various elements depending on the configuration (GPT2Config) and inputs.

Example of single-label classification:

>>> import torch
>>> from transformers import AutoTokenizer, GPT2ForSequenceClassification

>>> tokenizer = AutoTokenizer.from_pretrained("openai-community/gpt2")
>>> model = GPT2ForSequenceClassification.from_pretrained("openai-community/gpt2")

>>> inputs = tokenizer("Hello, my dog is cute", return_tensors="pt")

>>> with torch.no_grad():
...     logits = model(**inputs).logits

>>> predicted_class_id = logits.argmax().item()
>>> model.config.id2label[predicted_class_id]
...

>>> # To train a model on `num_labels` classes, you can pass `num_labels=num_labels` to `.from_pretrained(...)`
>>> num_labels = len(model.config.id2label)
>>> model = GPT2ForSequenceClassification.from_pretrained("openai-community/gpt2", num_labels=num_labels)

>>> labels = torch.tensor([1])
>>> loss = model(**inputs, labels=labels).loss
>>> round(loss.item(), 2)
...

Example of multi-label classification:

>>> import torch
>>> from transformers import AutoTokenizer, GPT2ForSequenceClassification

>>> tokenizer = AutoTokenizer.from_pretrained("openai-community/gpt2")
>>> model = GPT2ForSequenceClassification.from_pretrained("openai-community/gpt2", problem_type="multi_label_classification")

>>> inputs = tokenizer("Hello, my dog is cute", return_tensors="pt")

>>> with torch.no_grad():
...     logits = model(**inputs).logits

>>> predicted_class_ids = torch.arange(0, logits.shape[-1])[torch.sigmoid(logits).squeeze(dim=0) > 0.5]

>>> # To train a model on `num_labels` classes, you can pass `num_labels=num_labels` to `.from_pretrained(...)`
>>> num_labels = len(model.config.id2label)
>>> model = GPT2ForSequenceClassification.from_pretrained(
...     "openai-community/gpt2", num_labels=num_labels, problem_type="multi_label_classification"
... )

>>> labels = torch.sum(
...     torch.nn.functional.one_hot(predicted_class_ids[None, :].clone(), num_classes=num_labels), dim=1
... ).to(torch.float)
>>> loss = model(**inputs, labels=labels).loss

GPT2ForTokenClassification

If past_key_values is used, only input_ids that do not have their past calculated should be passed as input_ids.

Only Cache instance is allowed as input, see our kv cache guide. If no past_key_values are passed, DynamicCache will be initialized by default.

A TokenClassifierOutput or a tuple of torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various elements depending on the configuration (GPT2Config) and inputs.

Transformers

GPT-2

Notes

GPT2Config

class transformers.GPT2Config

GPT2Tokenizer

class transformers.GPT2Tokenizer

save_vocabulary

GPT2TokenizerFast

class transformers.GPT2Tokenizer

GPT2 specific outputs

class transformers.models.gpt2.modeling_gpt2.GPT2DoubleHeadsModelOutput

GPT2Model

class transformers.GPT2Model

forward

GPT2LMHeadModel

class transformers.GPT2LMHeadModel

forward

GPT2DoubleHeadsModel

class transformers.GPT2DoubleHeadsModel

forward

GPT2ForQuestionAnswering

class transformers.GPT2ForQuestionAnswering

forward

GPT2ForSequenceClassification

class transformers.GPT2ForSequenceClassification

forward

GPT2ForTokenClassification

class transformers.GPT2ForTokenClassification

forward