Scribd
Upload
2 views12 pages

Seq 2 Seq Model

The document discusses the Sequence-to-Sequence (Seq2Seq) framework in Natural Language Processing (NLP), which is essential for tasks that require transforming one sequence into another, such as machine translation, summarization, and dialogue generation. It outlines the core components of a Seq2Seq model, including the encoder and decoder, and explains the training process, including the use of teacher forcing and attention mechanisms. Additionally, it presents a practical example of building a student support chatbot that translates English messages into Hindi using a small parallel dataset.

Uploaded by

nizamuddin00128
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF or read online on Scribd
2 views12 pages

Seq 2 Seq Model

The document discusses the Sequence-to-Sequence (Seq2Seq) framework in Natural Language Processing (NLP), which is essential for tasks that require transforming one sequence into another, such as machine translation, summarization, and dialogue generation. It outlines the core components of a Seq2Seq model, including the encoder and decoder, and explains the training process, including the use of teacher forcing and attention mechanisms. Additionally, it presents a practical example of building a student support chatbot that translates English messages into Hindi using a small parallel dataset.

Uploaded by

nizamuddin00128
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF or read online on Scribd
5/18/26, 9:52AM student_copy nip. NLP_seq_to_seq.pynb - Colab Theory - Sequence-to-Sequence Frameworks. So far, we have studied models that mainly do one of the folowing: + assign probabilities to sequences + classify sequences, + represent words or sentences as wectors But mary NLP problems requie something more powerful: transform one sequence into another sequence ‘hiss the setting of sequence-to-sequence learning, often written as Seq2Seq. (Online visualization reference - bitos:/alammar github ioNsualzing-neural-machine-translation-machanics-of-seq2seq-madels- with-attention) Why Seq2Seq Is Needed ‘large number of NLP tasks are naturally ofthe for: input sequence -» output sequence amples include: + machine transation “exam schedule released” > translated sentence in another language + summarization long complaint -> short summary + paraphrasing formal toxt -» simple text + sialogue / response generation query — reply + grammar correction incorrect sentence -> corrected sentence ‘classification model is not enough here, because the outputs nota single label We need a model that can produce a variable length sequence as output. Core Seq2Seq Idea ‘Abasie Sea2Seq model has two parts: Encoder Reads the input sequence and converts it nto an internal representation, Decoder Uses that represen nto generate the output sequence token by token. ‘so the overall ow i: Bacar Deooter 2, 2)--)27 ———> context representation > yiy¥iy---sUr! where: + 21,-..,27 are the input tokens ntips:ifcolab. research [Link] CHROUsZUQK'320IVFYv47GLxGGi343007usp=sharinglscrolTo=-hKyb_ATxNDSprin’Mode=true we, 5/18/26, 9:52AM student_copy__nip_§_NLP_seq_to_seq./pynb - Colab + ysy op" ate the output tokens + Tand 7" may be diferent > ARunning Example | Zell hicden Encoder Mathematics ‘Suppose the input sequence is sch token is converted into an embedding: ex ~ Embedding) the encoder processes these embeddings on step at tine. fw use an RNN-stle encoder te hidden state f updated as: = len he) where: + eis the embedding of the current input token + eis the previous hidden state + J may be an RNN, LSTM, or GRU update ‘Alter reading tho fllinput sequence, the encoder produces final hidden representation ‘A simple basic Seq2Seq model uses: e=hr asthe context vector. This context vectors supposed to summarize the input sequence. » Decoder Mathematics ‘The decoder generates one token at a time Let the decoder hidden state at step ¢ be s. ‘The decoder update can be written as: where: + yp isthe previous output token + 5,1 is the previous decoder state + cis the encoder context vector ‘Then the decoder computes output probabilities: Plye | yts-+-4ense) ~ softmax( Wes, +b.) ‘So at each step, the decoder predicts the next output token from its eurrent hidden state YY ="once upon a time" y= vector("once") y.t = vector("upon’) <= st (hidden state) <= yt (Yonce)” train_set_pair = ("42 ae 1 Taf, “once upon a time") Since “once" is defintely the correct translated output. it can be used to force the model (Decoder RNN) to learn the correct translation However, ifthe same output is generated forthe test set, then we cannot be sure that “once” is the correct output. Therefore, using ‘once" fer the prediction of the next token “upon during testing is not that reliable. ‘Assume that during training forthe following par train_set_pair = ("¢e> are wt arf, “once upon a time") ntips:ifcolab. research [Link] CHROUsZUQK'320IVFYv47GLxGGi343007usp=sharinglscrolTo=-hKyb_ATxNDSprin’Mode=true aire 5/18/26, 9:52AM student_copy_nip_5.NLP_seq_to_sea.ipynd - Colab the actual generated output by the Oecoder RNN as the fst token is “ine”, “nine upon a time” Should the output “nine” and its corresponding value be used in the calculation the output “uoon"? fon when y.t-1is “nine”, we will not use it for the ealeuation of st Instead what we willuse is “once” as y¢ TRAINING, But this all only during During testing, the previous generated token willbe directly used forthe calculation of stasis. v_ Start-of-Sequence and End-of-Sequence Tokens ‘To make sequence generation possible, we use special symbols such as: During decoding + the first input tothe decoder is usually + generation continues until the model outputs ) ‘So the decoder learns to generate output as @ sequence that begins and ends explicit, » Training Objective IF the correct ourput sequence is: vive then the conditional probability ofthe wale output sequences Plyisvay-- sur |a1,22)--,27) = ]] Pye | yiy---syease) “Taking logs gives: Jog Plyts-+-s¥0r | 215-27) = Slog Ply | viy---sye-25¢) So training typically minimizes the negative log-Ikelhood: L= Slog Plve|vig-v-ste-006) > Teacher Forcing 14 dlls hidaon > Inference vs Training \y Bealihidon » English + Hindi Seq2Seq with Attention for a Student Support Chatbot This section of this notebook bulds a small neural machine translation demo for a campus student-support setting v Use case ‘The chatbot should translate short English student-supp 1essages into Hin, such as: ntips:ifcolab. research [Link] CHROUsZUQK'320IVFYv47GLxGGi343007usp=sharinglscrolTo=-hKyb_ATxNDSprin’Mode=true aie 5/1826, 952M student_copy_nip_6 NLP_seq.to_sea/pynb + ee receipt uploaded” —+ "apes we srt gE + oxam schedule released — “war pra aa + *hostel room allotted” — “orarra ra sete GT » Model idea We use an encoder-decoder with attention: + the encoder reads the English inp + the decoder generates Hindi one token at atime + the attention mecha sentence » Important training vs inference difference Training We often use teacher forcing: the decoder receives the correct previous Kin token Inference ‘The decoder does not know the correct next token. It uses its own previous pre ¥ Core equations IF the English input tokens are: the encoder produces hidden states: Pasha hr [At decoder step t, we compute an attention score overall encoder hidden ‘The attention weights are aug = softmax(score(s.-1,hi)) we + 8:1 is the provious decoder hidden state + hy is the encoder hidden state for input position ¢ ‘The context vectors: = Loawhs ‘Then the decoder predicts the next token using its hidden state and the context vector ibrary Imports Lngort rumpy as 9p Angort torch. nn. functional 36 F ‘from [Link] inpor® ataset, atateader 1 Sot the seed to 2 xed murder for reproductosiisy [Link](st#0) noveandon-se0d(S torchunanualseed(SEED) device = torensdevice(“cusa" 1¢ torch. cada 4s avatlable() else “epe") print(“Using devices", sevice) hntips:[Link] google. comidrvel CHROUsZUQK’32QIVFYv47GLxGG343007usp=sharingtscro Te Cola sm scores all encoder hidden states at each decoding step and bulls a context vector Kyb_ATxNDSpriniMode=true ana 5/18/26, 9:52AM student_copy nip. hntips:[Link] google. comidrivel CHROUsZUQK'32CIvFYv47GLxGGi343007usp=sharinglscrolTo=-hKyb_ATxNDSprin’Mode=true Using device: cpu CPU runtime would be suficient for the small demo. Ne GPUs are needed NLP_seq_to_s0q.pyn - Colab We include short phrases and slg Step 1: Create a small parallel English-Hindi dataset ‘This is a toy dataset tallored to a student-supportchatbot. longer variants so the model can learn: + fee-related language + exam-related language + hostel-related language + portalhelpdesk language Because the dataset is small, we will epeat the examples several tines during taining pates = [ foe receipt uplosses”, "Yo whe ates ge), fee receipt verifies”, "Ys We ets GE) foe payrent successful", “RSP HRI a BSH"), {oe pms oedig EE TB, (r#ee portal open" ee (Cice portal errr, ages See 4 gb, r c payment receipt watlable, “yn xed sueal 8), receipt not visible", "The Rien wl 2 a@ By, Coxon senedule velensed”, “atl arden of 5"), (Coram tinetable uploaded’, Tha a A URNS BE, exon date announced", "hl FAR RET HE"), (Chall tieeet svatiable", "B2R0 TE uel ty, (Conan result dectaree”,” ad ofeord BRU EI"), (Covan form submittes”, GG 1H oe ga, hostel room 9 ¢ exec", "OAT AR TT BN), (hostel atlotnent List publishes", "OTs steal gH wets FP), (hostel application submitted", "SHIRE cy SM EK"), r c hostel room available”, "BIsKG@ @R! weal 2°), hostel status upeated”, "Brae FUR ote ge), (scholarship notice releases", “oral Ge ol BE), (cteholarento application subgécsed", "BRY® S122 GMT BS"), (portal status upeatea", "Ses FRA ake BE), c c Student query received”, "BIH WN HT BI), Support request subnieeed”, “BEM NW ot GAN"), your fee receipt has been uploaded", rama) Yew wee oats BE your fee receipt Ras been verified, "ST Hep wh were Bf the ex Stele fas Been released, “eel SHE tt $57 *, the fertel allotaent ist has bee published", “OHaG orca a wata BE #, {Cyour hostel roon has been eliotted", "SINGH BART GT MEG EON) (Clhe snot sranap notice na been selena» “ORGY Ge Og) >, Step 2: Tokenization and vocabulary ‘We use simple whitespace tokenization (works only for this demo, as discussed in previous classes). We also add special tokens: + (epad>)for padding + cunk> for unknown words + | for start of sequence + / for end of sequence SPECIAL_TORENS = ["", "eunks", “es08>", “ce0s>"] ef tokenize( tent) return tont-lowor() strip() sp2it() siz 5/18/26, 9:52AM 4 busta vocabularies sre_voesd = set() ‘eelvoceb = et() for src, tt in pairs: Sre_woess update( tokentze(sre)) {tgtavocad update tokentze(eet)) src_stos = SPECIAL_TOKENS + sorted(sre_woca®) ‘geitor = SOECTALLTORENS + sorted(tgt_vocad) sre_stot = (tok: 4 for £, toe An enumerate sre $t08)} 22) egestas = (toe: for £) toe J ern ce 00_sRc = sre_stost") Pav 161 = tgt_stoif“") UUs or = eetastei["", “", “algtnent’, “atlotted’, ‘announced’, "application", ‘avatlabie', Sanple target vocab: ["", “ano”, "ss", “eos”, SEN, “ “des et encode_source(text) student_copy ip. NLP_seq_to_seq.pynb - Colab return [ste_stel.get(tok, UNKSRC) for tok An tokentze(text)] et encode target(text) ‘return (508-T6T] + [tet_stod.get(tok, WM _TGT) for tok in tokenize(text)] + [£05 161] ‘encoded pairs = ((encode_sourcasre), encode target (tet), src, tet) for src, tet in pairs] mnax_sre_len = max(len(encode_source(sre)) for src, _ in pats) ‘gl tn pairs) raxctgtilen = max(lenencode_target(tgt)) for, print(“hax source lengths", max_sre_len) print (“hax target length:", maxtgt_len) toe source Lengths 7 ox target Length: & Step 3: Train/validation split and dataset ‘Assume the train-validation split ratio as 85:1. Angort rancon ‘expanded + encoded patrs * 62 ¥ bate Augrentation random. shut eCexpanced) split = 2nt(0.85 + len(expanded)) ‘rain data = expanded[ split) val. data = expandea[spiit:) class Iranslatiowatasee(Oxtaset) def inis_(self, [Link] = te self imax srclen + 9ax_s Selfinae_tgt_ien = mack ler det _ten_(se4) Feturs Ten(seitsata) tips:[Link] google. comidrvel CHROUsZUQK'32QIVFYv47GLxGGi343007usp: sta, eax_sre_len, wax_tet_ten) We create a sightly larger toy training set by repeating and shutfing the examples, aringtscroTo=-AKyb_ATxNDSprin’Mode=true ez 5/18/26, 9:52AM student_copy__nip_§_NLP_seq_to_seq./pynb - Colab dof _getiten_{seif, 10): Sreids, TRL Acs, sre_toxt, tee toxt = [Link] ich] sre_pad = sre_tds + [sre_stot{"éoad>"]] * (self.max_sre_len = Len(sre_tds)) tgt_pad = tgi_tds + (tgt_stol[cpad>"]] (selF.maxctgt_len ~ len(tgt_tds)) return ( ‘orch-tensor(sre_pad, étypestorch-20n#), orch-tensor(2gt pad, étype=torch-20n—), tgtltext, ) ‘ran_ds = Translationoataset(tratn dota, max_see_leny 9% val_45 = Translationgataset(val_data, max_sre_ien, nex tet_len) ‘rain toader = oatatoader(train ds, baten st valtoader = atatoader(val-ds, batch sizaci6, shut print(“Train size:", Len¢erainds)) print(val size:", len(val_es)) train size: 3520 val size: 270 ¥ Step 4: Encoder ‘The encoder reads the English sentence and returns: + all encoder hidden states + the final hidden state \We keep all encoder outputs because attention needs them at decoding time, class Encoderstu([Link]) def _inis_(Self, input_vocab_size, evbed_din, iden din): Biper()-_inst_0) [Link] > [Link](ingut vocab size, ended dia, padding texsre stos["*)) self gra = [Link] din, den di Baten. FirsteTeue) et foruara(senf, sre) f sre: [batch, sre_len} Cnbedded = self .enbedding(sre) 4 (atch, sre_ten, enb_din] fencoder_cutputs, hidden = [Link](enbeddes) & encoder outputs: [baten, see_ten, hidden din) |W middens [, Baten, header ae] return encoder outputs, hidden ¥ Step 5: Decoder The decoder takes: + previous target token + previous hidden state ‘and predicts the next token. class Decosertu([Link]) def _intt_(se1f, vocae_size, enbed din, hide sin) Siper()-_init_O [Link]éing > [Link]éing(vocab_size, enbed din, padsing_1ds-0) [Link] = [Link] dia, aden_din, baten_virstetrue) [Link] = [Link](hidden-ain, vocab size) et forward(se1f, input_token, tegen) # inpet_token shape: [batch] {nput_token = input [Link](1) & [bateh, 2] fenb = self-enbeccing(input token) [8ateh, 1, embed din] output, hidden = sel¢.gou(anb, Mden) ogits = seif-fefoutout-squeeze(s))—# [bateh, vocab size} return logits, Ridden htips:fcolab. research google. comidtvel CHROUsZUQK’32CIVFYv47GLxGGi343007usp=sharinglscroe Kyb_ATxNDSpriniMode=ttue m2 5/18/26, 9:52AM student_copy_nip_5.NLP_seq_to_sea.ipynd - Colab » Step 6: Seq2Seq Wrapper ‘This wrapper combines encoder and decoder and supports teacher forcing during training class seqzsea([Link]) det init__(self, encoder, decoder, devicesrepu*) Siper()-_intt_O self secoder = cecoder [Link] » device det formra(selt, sre, te nize = are,size(@) max decode_len = [Link](3) se outputs = torch 2eree(batch_eize, nax_decods_lon, vocah_size, [Link]) + Maden = sel¢encoder(sre) snput_toten = toren. fu (eaten size), [Link] > ‘for in range(nux_cecode_2en) logits, hidden ~ self-decoder(input_token, hidden) covtputst:, t, £] = logits Logis. argnax(2) 4 {gt 45 not None and [Link]() < teacher forcing ratio fnput_token = tgtl sy t] return outputs ¥ Step 7: Training Utilities encoder = EncadersRuten(sre_itos), enbed_dine32, hidden din64) ovcoderGRufien(egt ites), enbed_dines2, hidden dine) seq2seq model = SeqaSeq(encoder, decoder, device-device),to(device) criterion = [Link](ignore_index-PAD_T6T) loptinizer = [Link](seq2seq model paraneters(), ©) et evaluate seqzseq_loss(nodel, loader): fodet eval vesth torch.no_gradt) for sre, Ugly sre_text, tpt text tn leader sre, (gt = [Link](devies), tpt. to(deviee) outputs = nodel(are, tetetet, teacher [Link]®.0) vocab size » outputs. size(-1) toss = criterion([Link](-t, vocab size), [Link](-1)) fotal_toss 4= loss. sten() * [Link](0) totalLitens r= srecsize(®) hntips:[Link] [Link] CHROUsZUQK'32CIvFYv47GLxGGi343007usp=sharinglscrolTo=-hKyb_ATxNDSprin’Mode=true ane 5/18/26, 9:52AM student_copy__nip_§_NLP_seq_to_seq./pynb - Colab sl_loss / total_atens seq2seq(rodel, train loader, val_loader, epoehse25) history = ¢ “teain-toss*: [e wal toss": 1) > snstal_val_loss = evaluate seq2seq_loss(nodel, val_teader) for epoch in range(epochs) novel train()| sre, tgt'= sreto(device), [Link](device) optinizer.zero_prad0) outputs = nodel (sre, tet-tet, teacher forcing ratio-8.7) vocab_size = outputs. stze(-t) loss 5 [Link](-1, vocab size), [Link]-1)) toss. baclnara() [Link]() otal_toss += [Link]() + [Link](0) total_itens += are-size(@) val_loss = evaluate_seq2seq_loss(nodel, val_loader) append(train_1055) append(vat_tosa) f opoen X'S == @ oF epoch == epochs = 1 Printte"Epoch (ep0eh=76) | Train lose = (erain_losst 46} | val return history, initial val_toss ¥. Step 8: Greedy Decoding for Inference oes = (val_loss:.40)") er greedy eacode(voeel, [Link]() oR, Wa Tea 05 = encode source(text) = de + [PAD_SRC] * (0 fre = [Link]( {sre ids], typ len ~ Len(see_ids)) Torch long, devicendevtee) vesth torch. no gra) outputs = nodel(sre, tgteNone, eacher_forcing_ratiors Ads = outputs. argnax(-1)-squeeze({)-tolist() words = 01 for sex in predicts sas token = tet itostiéx] 5 oken not in {Mepads*, “es08>") words -sppen(saken) return " " Joingwores) nax_Len-nax_len) » Step 9: Before Training Predictions aeno_inpats = T your fee receipt has been verified", he hostel allatacet list hae been publish tips:[Link] google. comidrvel CHROUsZUQK’32QIVFYv47GLxGGi34300718: Kyb_ATxNDSpriniMode=true oz 5/18/26, 9:52AM student_copy_nip_5NLP_seq_to_sea.ipynd - Colab print("tefore training”) print("anpyt_:", text) print(“outost:", greesy decode(seq2seq nodel, text)) prane() autpec: Hea ee one eh he wre AES are ge Re we oI outpue: Gor Read aes onal Qe le Wels ache aehe sets sets ure Goa Rnd gue owe PP she owas ous hs orbs ames ¥ Step1 Train the Seq2Seq Model history, initial_val_loss = train, seq2seq(seqzseq_sogel, train, der, epochs=20) 1 NOTE: GPU runtine nay be required after naking wodiflcations in the nunber of epochs / 4 rodel architecture and / or dataset and / ar other training configuration parancters 4 anste solving any aesigmant later based on this inp print("tnitial validation 1oss:", round(initial_val_loss, 4)) print("Final validation 1oss:*, Found(hstory["vai_oss")(-1], 4)) Epoch @ | Train Joss = 0.6585 | Val loss - 0.6327 Sheen | tain tot «6.807 | Wa Lat «ce Epoch 15 | train toss = 0.0601 | Vat loss = Epoch 2 | Train Joss = ece00t | Vat Joss = initial valddation Jess: 4.046 Final validation loss: 0.0082 ¥ Step 11: After Training Predictions nt("Prectetions AFTER training") print *y {for text in dena inputs: prine(“anput =", text) print (“output”) greedy. print) tex) outpue: Shier ion ara gore outpee: Se ae eel wets GG ¥ Assignment [As can be seen inthe output ofthe code below, wien unseen pars (test data) are used to evaluate the model that has been trained earlier, correct translations are not learnt. Ty to identify the notion that isnot being learnt by the model Without ctl adding any of the pai ‘make other minimal changes to the dataset or preprocessing techniques in the notebook, to ensure thatthe English-Hinai translation of the following pairs is also correctly done by the newly trained model. You may need to use GPU runtime for this training from the givon generalization test_pairs to the trai (Cthe exon schedule has not been released”, “tte orm ont = frm ma 3°), (the ea tinetable has not been uploaded”, "ln ama areate =e ME B+), (the exan form has rot been approves", elit td ena af Pra sar 8), htips:[Link] [Link] CHROUSZUQK'3zQIVFYW47GLxGGi343007usp=sharinglscrolTo=-AKyb_ATXNDSprin’Mode=ttue 10/12 5/18/26, 9:52AM student_copy nip. (the hall ticker ts not avaslable", "WAR GH GusaI Tet 8), (the ean portal has not been opened”, “Sten ee a Stet Tart), (the exae notice has aot been pualished”, “Then ae were at at rE By, (the ean detaiis are not avatianie®, “Waen awl sweat =E 2), (the xan status hat nat been upasted”, "Uday (Re aride aQh aig B+), (the senedute i not visible", “ani fawre ef 2 i By, (Cory roll nunber is 232959800", "Ra ae wT VKATCICG By 1 4 Add your ou roll number in the Last sanple of the above generalized test set. print(“Presictions on generalization sest_pairs AFTER snitial training”) print(~ *) ‘for text, translated text in generalization test_pates: print(“Teput 2", text) print(“output:", greedy decode([Link], text)) print() gation related Predictions AFTER trainin Input + the exax, schedule hag not been released ‘output: Stay eR Te gt ope + ua nies gt er wend input: the exan fore has not Seen approved outpue: aT SE gt Input + the exax, result has not been ceclared utpue: lat dan td) gt Input + the hall ticket is not ovatlable output: her aaa outputs et tft ee ga Input + the oxen noties has not been published output: wre GT ew Inpet + the exge details are not available foutpue: lat UOT Rar gal cin te ex tus fs pt be ped Sepa: Oana a one ge output: SH AEH EP GO NLP_seq_to_sea.pynb - Colab (the xan result has not been declared”, "atlan wra sts Hf oer mar Be), v What to Observe Before traning: + outputs are random, repetitive, or empty ‘+ the model does nat yet know how to map input sequences to output sequences After training: + outputs become more meaningful ‘+ the mode! learns the rewrite mapping + loss decreases This demonstrates the key idea of Seq2Seq the model learns to transform one sequence into another sequence Important Interpretation In this basic Seq2Seq model, the decoder relies on the final encoder hidden state as the summary ofthe whole input hntips:ifcolab. research [Link] CHROUsZUQK'32CIVFYv47GLxGGi343007usp=sharinglscrolTo=-hKyb_ATxNDSprin’Mode=true wwe 5/1826, 952M student_copy_nip_6 NLP_seq.to_seaipynd - Golab Soif the input becomes much longer or more comple, the fixed-size context vector may become insufficient ‘That limitation motivates the next major improvement: + Attention because the decoder should ideally be able to look back at diferent pars ofthe inout, rather than depend only on one compressed vector, htips:fcolab. research google. comidtvel CHROUsZUQK’32CIVFYv47GLxGGi343007usp=sharinglscroe Kyb_ATANDEprinMode=ttue 12/12

You might also like