Scribd
Upload
3 views21 pages

Bert t5

The document provides an overview of BERT (Bidirectional Encoder Representations from Transformers), detailing its architecture, pretraining methods, and applications in natural language processing tasks such as text classification and question answering. BERT utilizes a masked language modeling approach and next sentence prediction to learn contextual representations of language, allowing it to understand the meaning of words based on surrounding context. Additionally, it explains input representation, special tokens, and the fine-tuning process for specific tasks.

Uploaded by

nizamuddin00128
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF or read online on Scribd
3 views21 pages

Bert t5

The document provides an overview of BERT (Bidirectional Encoder Representations from Transformers), detailing its architecture, pretraining methods, and applications in natural language processing tasks such as text classification and question answering. BERT utilizes a masked language modeling approach and next sentence prediction to learn contextual representations of language, allowing it to understand the meaning of words based on surrounding context. Additionally, it explains input representation, special tokens, and the fine-tuning process for specific tasks.

Uploaded by

nizamuddin00128
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF or read online on Scribd
510126, 6:08 PM ‘student_copy_rip_7_NLP_BERT_T5_GPT.ipynb - Colab » Theory » BERT: Architecture, Pretraining, and Core Concepts References: 1. Base Paper: hitasiiansvora/adt/t810,04805 2. HuggingFace: htipsv/[Link]/transformers/model_dociber ERT stands for Bidirectional Encoder Representations from Transformers. Itis an enceder-only Transformer model designed to learn deep contextual representations of language by using both left and right context at every layer. ‘This makes BERT especially effective for language understanding tasks euch asi + text classification + sentence-pair classification + token labeling + extractive question answering The central idea behind BERT is that the meaning of a token depends on the words before it and after it. nstead of predicting the next token ina left-to-right manner, BERT is pretrained msinly through Masked Language Modeling (MLM), where selected tokens are hidden and the model predicts the original tokens from context Core Idea of BERT In many sentences, understanding one word requires looking at both the lft and right context Consider the sentence: ‘The dean approved the _after he committee meeting To infer the missing word, both sides ofthe blank ae useful. BERT s designed to learn inthis way. ‘left-to-right language model predicts the next token using only previous tokens, BERT instead learns contextual representations by allowing each token to use information from both directions. ‘This is why BERT is mainly used for understanding text, not for long-form free generation, ¥ Input Representation in BERT ‘The input to BERT is formed by surnming three embeddings for each token positon: 2, = Eicten(i) + Bregment(i) + Bposition i) Hore: ++ Exoken(i) represents the token identity + Baegnens (a) Fepresents wether the token belongs to sentence A or sentence 8 + Byositon(t) represents the token's position inthe sequence 0, for each input position, BERT knows: + wat token is present, + which segment it belongs to, ‘+ where it occurs in the sequence. ¥ Special Tokens BERT uses special tokens ints input sequence, htips:ifcolab. research [Link] Ry-BOADIS8rSm (pleces)") playing > (playing’] UnbeLLevabie 3 [ronal tevabie") ontertualization > [ restructured >I » Encoder Stack BERT s built from stacked Transformer encoder layers. sch encoder layer contains: + multi-head self-attention + feed-forward neural network + residual connections + layer normalization Since BERT is bidirectional, each token can attend to tokens on both sides oft inthe sequence, ‘This allows the same word to receive different contextual representations incifferent sentences. For example, the word bank has different meanings in “+ He sat near the river bank. ‘+ She went to the bank to deposit money. BERT can produce different contextual vector fr the same surface word depending on the surrounding context. Masked Language Modeling (MLM) ‘The main pretraining task of BERT is Masked Language Modeling, ‘Asmall portion ofthe input tokens is selected for prediction, Inthe original setup, 15% of WordPiece tokens are selected ‘Among those selected tokens: + 80% are replaced with [was] + 10% are replaced with a random token + 10% are left unchanged ‘The model must predict the orginal token atthe selactd positions. If the orginal sentences: ‘The operating system manages memory and processes. Cone masked training version could be: ‘The operating system manages [MASK] and processes. The targets stil the original wore: memory ‘The model uses the contextual representation at the masked position to predict the original token. For a selected position , lt the final contextual vector beh ‘The model uses h; to predict 2 probability distribution over the vocabulary: P(token at position é | corrupted input) htips:ifcolab. research [Link] Ry-BOADIS8r8m<7UBBpCxIMusGUHNcoB 75 aringiprinMode=true 321 510126, 6:08 PM ‘student_copy_rip_7_NLP_BERT_T5_GPT.ipynb - Colab htips:[Link] google. comidrvel Ry-BOADIS8r8meZUBBpCxIMusGUHNcoB 75 Teaning minimizes cross-entropy loss between the predicted distribution and the original token identity. ‘This setup allows BERT to learn strong contextual language representations. Jip -9 snstall torch rom transformers saport pipeline lonasker = pipeline("fill-nask", nodel-"google-bere/bert-base-uncased”) xanples = “The operating systen manages (MASK) and processes." a compiler translates source code into (MASK) code.” “The deen approved the [MASK] efter discussion, for ex in examples: prine¢*\nIW@uT :*, @0) futputs = unmasker(ex, top_ke5) for 0 in outputs; prine(#"{oltoken_ste*]:322) -> {of score" :.46)") rode sattonsors: 100% acs (00060000, NEL) Leadng weighs: 100% {20a702 2000<0000, 857.80, Matrisng parc pects ansorm sense. SereFarkaskedl§ LOAD REPORT fran: google-ert/bert-basemuncased wey Vets || [Link] | wNexPecTeD || Cislsequrelationsnip bias | URBPECTED || [Link] | UNEXPECTED || [Link] | UNEXPECTED || “TWNOGECTED scan be ignored when Loading fon aitferant task/architecture; not ok Lf you expect Léeatical arch, INPUT + The operating system manages [MASK] and processes prograns -> @.0836 devices > 6.0667 dats => 8.984 systens “> 0.0606 INPUT + A conpiler translates source code Into [MASK] code. machine "> 0.3695, progran => @.0328 output => 6.0296 compiled => 8.0275 INPUT 1 The dean approved the [WASK] after discussion, ‘proposal => 8.1528 plan -> 0.2806 request -> 2.2465 ‘idea -> ovease ‘enange => 0.0827 Why the 80/10/10 Strategy is Used IF every selected token were always replaced by [MASK], then the model would become too dependent on seeing [MASK] during training However, during downstream fine-tuning and real usage, normal text usually does not contain [ASK] To reduce this mismatch: ‘+ most selected tokens are replaced by [ask]. + some are replaced by random tokens, + some are left unchanged. ‘This encourages the model to learn robust contextual reasoning rather than relying only ona special masking token. snport rancen vocab_sanple = ["banana”, “server”, “necting” the", "dean", “approved, “the, “revises”, “proposa2") “buaget", “report") waringtpriniMode=true a 510126, 6:08 PM ‘student_copy_rip_7_NLP_BERT_T5_GPT.ipynb - Colab htips:ifcolab. research [Link] Ry-BOADIS8r8m<7UBBpCxJMusGUHNcoB 75 reture "(MASK “return randon-chotce(vocab_sauple) een( token) ancon() chosen_index = 4 # token: “revised corrupted = [Link]() upted[chosen index] = corrupt_token(tokens chosen. index) print("Original :", tokens) rint("corrusted:", conrupted) print(-Target:", tokens[etosen_tndex)) original : ['the', ‘dean’, ‘approved", ‘the’, ‘revises', “propesal') Corrupted: ['the', ‘dean’, "approved", ‘tre", “[HASK]", ‘proposal’ ) Next Sentence Prediction (NSP) In the original BERT pretraining setup, SERT is also rained with Next Sentence Prediction. In this task, the model receives two text segments: + sentence A + sentence 8 ‘The model predicts whether sentence Bis the actual next sentence after sentence A, So the output is one of two labels: + aNext + NotNext ‘This helps BERT learn relationships between text segments, which is useful for tasks involving sentence pais. ‘Atypical BERT sentence-pair input looks lke this: CLS] Sentence A [SEP] Sentence 8 [SEP] lore + [cL] provides 2 sequence-level representation + |[SeP] separates the two segments + segment embeddings indicate whether a token belongs to sentence A or sentence B Output Representations of BERT [At the end ofthe encoder stack, produces a contextual vector for every token position in the sequence. These outouts can be used in two main ways Token-level use Each token vector can be used directly for tasks such as + named entity recognition + token classification + extractive question answering Sequence-level use ‘The final hidden state corresponding to [CLS] can be used for tasks such as: aringiprinMode=true 521 510126, 6:08 PM ‘student_copy_rip_7_NLP_BERT_T5_GPT.ipynb - Colab ‘+ sentiment classification + topic classification + sentence-pair classification ‘from transformers sapere Autotoded rode] = Autotiodel.ron_pretrained("google-bert/bert-base-uncosed", output hidéen_statessTruc) ‘text = “BERT sullds contextual representations of tokens. Anguts = vokentzer(text, return tersorse'st") with toreh.e_grad) lutputs = rodel(**inputs) last_hiéen = outputs. last_pidden state Iniddon_states = [Link] states print(“Last Madden state shapes", last nidden. shape) print(“Cts vector shape 7, Zast_nidden{:, 8 :).shape) print(*hunber of hidden states:", len(hidden_stetes)) Lesage: 100% arte [onca sets = 1 “the lecture was clear and engaging.” » the explanation was confusing and rushed.” 1 print(*\nT#xt =", text) Drane(*PRED :*, Classifier(text)) ct jan: “008 4477 (on on.0000, 1328) Prtoch_medelbn: 1008, 34394 (0008000, 120K) Leang weighs: 100% 201201 (000-0000, £3758, Mating paral ght rod satteear: 100% sags (0005000, 68.708) lehanaet anion: 100% ‘sata (ooc0