INDIAN INSTITUTE OF TECHNOLOGY | KHARAGPUR L [7 Siamp7 Signature ofthe ivigiator EXAMINATION ( End Semastor } | SEMESTER (Spring ) Roll Number Subject Number | C Deep Learning Important Instructions and Guidelines for Students 1. You must accupy your soa as per tho Examination SchedulerSiting Plan. ed off mode. 2, Bonet keep mabila phones or any similar electronic gadgets with you evon in the sit 2, Loose papers, clase notes, books or any such materials must not be in your possession, even ifthoy are relevant to the subject you are taking examination, 4, Data book, cades, graph papers, relevant standard tatlescharts or ary ater materials are allowed only when isttucted by the paper-setts, | + Uso ot insbuentbox, pra bo art eonpropanale clever Is ote ding th examinion, Hower, | atdetg ettentonsSranyeber pape (sean munton pap) owt part 6. wo on beth tes oft anova script nd rota ay page. Use ast pagel) ofthe answer eit for rough 1. ts your responsity ers thet you have sesh Atondance Shes Keep our Admit Carddonty Card onthe Seat cnosrg oy heer eae ans Barinaion at ne vege’ peed Song ar the srsunpo a ana of Bocas Sly probed he Eaton Pat i Slowed to ae aay he aswer Srp wh you Aare compat fu errnaon oo fet eae eat 10. Dung tho examination, ether inside or outside the Examination Hal, gathering information trom any Knd of sources of ‘exchanging information with others or any such temp willbe treated as ‘unfair means, Oo nat adopt unta means and o not indulge in unseemly behavior. Violation of any of the above instructions may read o severe punishment. ‘Signature of the Student To be Med in by the examiner Question Number} 1 | 2 3 fa] s e|7|a 9 | 1 | Tota Marks Obtained | ‘Marks obtained (in words) Signature of the Examiner ‘Signature of the ScrutineerCS60010 Deep Learning, Spring 2023-2024 26-Apr-2024, 02:00pm-05:00pm End-Semester Test ‘Maximum marks: 89 Instructions + Write your answers in the respective space provided in the question paper itself. Be brie? and precise. Please provide the final answer with related calculations. + Answer all questions. + There are no clarifications. Jn ease of confusion, youcan make a valid assumption, state that properly and proceed. {Questions start from the next page.1, How many weights and biases are there at each convolutional layer and fully connected layer in the VGG. architecture shown below? 8) If H Lash ! a vee z OF — Pago t of 11 —2. Consider a convolutional residual block that contains a batch normalization operation, followed by a ReLU activation function, and then a 343 convolutional layer. Ith input and output both have 512 channels, how ‘many parameters are needed to define this block? Now considera bottleneck residual block that contains three batch norealization/ReLUleonvolution sequences. ‘The fist uses a 1 x 1 convolution to reduce the number of channels from 512 to 128. The second uses a 3 x 3 convolution with the same number of input and output channels, The third uses a | x 1 convolution to increase the number of channels from 128 to 512 (see Figure below). How many parameters are needed to define ths block? o channels by channele by factor of four factor of four — Page 2 of 11 —3 (a) Consider the problem of predicting the nest word from a sequence of words using a fixed window neural language model (using the context of 4 words to predict the next word). Suppose, there are overall 50,000 words in your vocabulary, and each word is represented using a 300-dimension vector, Suppose the Ihidden layer has 50 dimensions, what are the total number of parameters that you wil have to learn? Ignore the bias terms for this problem, Cc) (b)_ Suppose you are using BE-LSTM for NER problem, and there are 3 NER tags: {PER,LOC,ORG}, and you are using BIO tagging scheme. Assume that each word has @ 300-dimension embedding, and the iden state for the forward and backward LSTMs are 200 and 100 dimensions, respectively. How many total parameters will need to be trained” Ignore the bis terms. as) (©). Suppose you are using BERT-base for topic classification of documents with 5 classes. How many task specific parameters will you need 10 use? @ — Page 3 of 11 —44. Suppose you are computing the word vectors using Skip-gram architecture. You have 10 words in your vocabulary, (hiyou, [Link],am,how, why, there, who, what} in that order and suppose you have the ‘window, hi there how are you" in your corpora. You use this window with ‘how" as the center word and two words before and after the center word as your context. Also, suppose that for each word, you have 2-dim in and out vectors, which have the folowing value at this point given as follows: Word | In-vector | Outvector HY G-) | C22 you | (1,2) | 1,2) they | 2,-2) we | (1,0 am | @) how | (i,-1) why | (1,2) there | (2,-1) who | 22) what | (1,1) ‘Table I: In and Out representations for words ‘What will the values at the input, hidden and output layer as per the Skip-gram architecture? What would bbe the total iss for this window? © — Pagedof 1 —5. Suppose, you are training your transformer decoder with the Ground trth: ‘how are you", The embeddings for these words are the concatenation of in and out embeddings showin in Table 1. Suppose you are using, tmulti-headed attention with f= 2. For both your attention heads, the first and third dimensions ofthe input define your query vector, and the second and fourth dianensions define your key vector. For the first attention hnead, the value is the in-vector, and forthe second attention head, the value is the out-vector. What will be the output of the mul-headed self-attention block for the word ‘are’ (before applying the projection matrix Wo)? You are using the sealed dot vector for self-attention. Ignore the start-of-sequence token. © —~ Page 5 of 11 —i search decoding for generation with a beam size of 10, By the fourth (a) Suppose you are using beam search decoding for gene! of 10. By the fourth step in the decoder, ie., when you are generating the fourth word in the sequence yy hypotheses for which you will have computed the log probability? (b) Suppose your vocabulary is of size 10. At the time of decoding, the probability distribution by applying softmax to the logits is shown below, How will this distcibution modify if a) you are using Top-k sampling with = 5, b) you are using Top-p sampling with p ~ 0.6? Please show the resultant distributions. ® 1 fo Vocab 4 “you [on thes [ot ere [on ‘an [ot ow [50 why [20 ‘here [30 “who [a5] what [0 (©) During BART pretraining, the sentence can be corrupted through “token masking’ and / or ‘text infiling’, among other possible methods. Use an example sentence, “I went to the store” to show that these objectives can lead to different inputs to the encoder. Which of these objectives is closer to TS pretraining? ‘Show the corresponding input and output for TS protraining. @ — Page 6 of 11 —7. Consider the ''S-Large pretsined model, which has a model-dimensionality of 1024, and a vocabulary size of 32k, Ut has 24 transformer layers, and has 16 atenton heads (or all different attentions). The feed- forward network has 4096 nodes. How many paremeters does ‘'5-Large has? @) (@) While trying c tain your LLM, you got 10 cimes more compote. You can use it to increase your model size by a factor of X, and/or increase your data (steps * batch sze) by a factor of Y. What would be ieal values for X and ¥? ® (b) Suppose you ate using (Vision) Transformer to encode images of dimensions 224 x 224 using patches of size 8 x8, Assume that the model dimensions are 512, and a Jeamable class token embedding is prepended. What wil be the number of learnable parameters forthe (a) positional embeddings, and (b) input representation? ® — Page 7011 —13? What (@) For the RLF taining, how isthe reward model Rif trained? Which model i ypiealy use ° data is required, and what is the loss function? (b)_ While training with poticy gradient, the weight update looks exactly like the normal cross-entropy loss, ‘weighted by the reward score. But is there any other key difference? @ (©) During RLHF taining, the reward function RS(s) is not exactly the reward model RMy(s), but has @ penalty term, What is the penalty term, ancl what isthe rationale behind using this? @ — Page 8 of 11 —(2) Suppose you are using QLORA to achiove 2-bit quantization with values: {—1,—0.2,0.6,1},and your input tensor is |-48, -3,38, 50]. What will you need to stove in the quantized format. Also, what will be the output after dequantization?: ® (b) Suppose you are using GPT3 forthe query auto-completion task, where the user types a prelix, and the system tries to complete this. You want to personalize this by fine-tuning individually for 10,000 users of the system. You have the following options: 1. You can use prefix-tuning at ll the layers with length of prefix as 4 and length of infix as 4 2. You can use LORA forthe query and key matrices at all the layers, with rank r = 2 3. You can use adapters (2 adapters per layer), with the projection dimension 2s 2 ‘Compare the number of parameters required for each of these options. o — Page Sof 11 —10. (a) Inthe VAE, the encoder predicts a mean and standard deviation, rather than just a latent veetor as in Autoencader. However, this may jest do local smoothing, with no continuity in the space. What is done to ‘ensure the global smoothness? @ (b) For the VAE model, derive the evidence-based lower bound in terms of the encoder and decoder parameters 0 and @. [Only derive the bound to maximize fog(Pr(x|9)], no need to simplify this.) @ — Page 100f ti —(©) In the diffusion model training, show that the reconsteuction term ~log[Norm,,{fi za, Gul 74] cam be expressed asa mean suited los to pred the nise using the deco. ® (@ In the diffusion model, why does the decoder model sample from Norotz, [Oy]? [Show that when the encoder is an forsufciently long number of steps, it converges to this @ — Page 11 of 11 —