NEURAL LANGUAGE MODELS The Architecture of Artificial Intelligence A Comprehensive Guide [EBEMEELMEaEieaeeg) t Modern NLP Fe OEDEFINING NEURAL LANGUAGE MODELS FON nC OR AONE Rares Rae ea on) Ce eC Ce Ue RR me core CC Cec RTs) Ce Re Oe ea ee ae cc Tec) PERE CT aL G ee eee Meter nT cise Mes tem REC NLYS Ce ee ee ea EC ce Cot oan tog represent a fundamental shift from rule-based linguistics to data-driven learning, where the machine learns the CCM ea acct aCe RaTHE PROBABILISTIC NATURE At their core, language models are probabilistic engines. They view language not as a set of fixed rules, but asa sequence of events where each word's appearance is conditional on the words that came before it. For example, if we see the phrase "The cat sat on the," a good model assigns a high probability to the word "mat" and a very OE SOOM et eco CCST cre UOC eect mm anasto Cee aL eR ee cme these probabilities across billions of parameters, the model creates fluent and coherent text that mimics human Dieta casTHE PRE-NEURAL ERA: N-GRAMS Before the rise of neural networks, the dominant approach to an eC cS CCR simply a contiguous sequence of ‘n' items from a given sample Cee eC These models worked by strictly counting how often word B followed word A in a database. While simple and effective for Dee Rog CCE CRU Re en raay Se CR CR eee eae Prepared by Sir Arshad JEU Wo ect ae RCo) assign it a probability of zero, making it incredibly brittle and Pino eee aten Cet amendBoel ee ae Peseta eT cc eet os At Sm Sc Can Rete cone Instead of storing massive lookup tables of word counts, neural networks attempt to learn a distributed Se a Cee a cane Ret ee cae ce PUR ac ERE Caen Oe cocci eT OCC CORO nT ect ch Mais Rae ncuritaik@l Prepared by Sir Arshad applies some of that knowledge to "puppy" or “canine” because their neural representations are similar. This, similar words. If the model learns something about "dog, ability to generalize is what separates modem Al from the rigid statistical models of the past.FEED-FORWARD NETWORKS (FFN) Suan eC Es Ola. r i Mts aoa d Erotica CMe nM TORT en Greet tety OC on aCe ecm eae a eee Cy Cee ROR Cot tae Cam LTS POOR eee ce Rn ean ccs SoC ee Te coo Tc itg Cn OR RCC ECCS Poa Sc eae could use vector representations, it stillTHE VOCABULARY PROBLEM CO ee es a ea Cee Ce RES CR ena ere terete Ae ETS approach is "one-hot encoding," where every word in the dictionary is a vector with one I’ and thousands of CECE AR aCe Gren ee te eS oR os ea Ca OS creates incredibly sparse and massive vectors that are computationally expensive. Furthermore, one-hot vectors Prepared by Sir Arshad [PCa is from “car.” This mathematical representation failed to capture any semantic similarity between related are orthogonal, meaning the vector for "apple" is just as different from "peat ConeseWORD EMBEDDINGS EOC CRC Ce ORCC eC d Sn es ee Cec mT Pr cae ORE SCR ac ee SO Ce eee Ree tT CO Rec eGR et Rose Cnc e Rt Rcoased eR Rae eM ect RTT ead together in the mathematical space. This was a paradigm shift Dee CCRC cme COM ILC Danny Pe en ec am nC e ny including apples, oranges, and bananas, all grouped togetherWORD2VEC & SEMANTIC ARITHMETIC TPE Meni Seti tes Ri Pa ee Ree CRE oo eee ee ecu a oe CON So CRS CeO RS Se cae CURR My Toe coco Ree model had learned relationships between concepts, capturing the "royal" attribute of King and the "gender" ce CR Me ROR ane soa Can enn Caw USE cee Tsao eRe ce eee eR eure Re ecu Cece ar acc cd neural network had organized the messy, unstructured data of human language into a structured, navigable Se Rca sRECURRENT NEURAL NETWORKS (RNN) To handle the sequential nature of SEN te cee eR ect Rec eee (AS CC Re Cae Co Ce OE CeCe aeons td OO eet Rc eC Meneses Breer Comer ame EY Cee CeCe This allows the network to process inputs Dec mC Taree nesTHE MEMORY FLAW: VANISHING GRADIENTS SE eco ae oe eee Ce enn CULTS eer eta ee nin te Cae eee ree eS Sc Rents Cone et ed De UC ACS Oe es Oe eR Cg Pon eR cro ees eevee Cena eT SEC ESt OSeaie Rosas CeCe SCL Sepa aan ear etter E reached the third, making it impossible to generate coherent long-form text.LONG SHORT-TERM MEMORY (LSTM) To fix the memory issue, the LSTM architecture was introduced. LSTMs are a special kind of RNN capable of ee See eee ee oe Rn ec ae eee gate, the forget gate, and the output gate. These gates actively regulate the flow of information, deciding what Cece eC eta aa ena Se ee ee Rn ed PCM cence ee ee ence CeeSEQUENCE-TO-SEQUENCE (SEQ2SEQ) Pes ee eat ea nee ee Tee On Cee aCe cay Encoder and a Decoder. The Encoder takes the input sequence (like an English sentence) and compresses it into A ee ee eee aaa RCE Pra USE cerca Rey nese ROM ee ener eer tea ay French sentence). This structure allowed neural networks to map sequences of different lengths to one another, Fe Paeaeed solving the problem that input and output do not always match word-for-word in erenceTHE INFORMATION BOTTLENECK Despite the success of Seq2Seq, a new problem emerged: the Information Bottleneck. The Encoder was forced to ee eu aa ee eee ee te a eer aS summarize an entire novel into one word; inevitably, details are lost. If the input sentence was long, the context vector would blur the nuances, leading to poor translations. The Decoder would struggle to reconstruct the realized they needed a way for the Decoder to look back at the source text.Si VEO Ue yi) ORO CR ne eee ect oe Poet a aS CMR nC me Cec an CMe CeCe Mechanism allows the Decoder to look at the entire sequence of encoder states. At every step of generating the output, the Ren tee eee ae oes Re eee Reece ye TRC ROR roc Cad Pea SORE gee cece Ceca Oe Tn a eco ReC Seo forth eaters Cyc a ete enn Coa UMS nad RES Cece enh"ATTENTION IS ALL YOU NEED" Feed Forward | f = Add & Norm |) Multi-Head Feed Forward Attention {Add & Norm | Hu Ke ‘Add & Norm Je -*(_ Add & Norm ———— ae ‘Masked Multi- ‘Multi-Head See. eo kere a oa ce Ris es SC Cad architecture, They proposed a radical idea: Creer ERNE ORSL Cee Oe coe aD Pen ae PCI Oia cae ORCS cS ban Rc aOR ced SOC cca parallel). This parallelization allowed forTRV) ee 48) Peco C Re Cae Cd occ COMA eC conan oc nd and output sequence, self-attention looks at the relationships between words within the same sentence. For CeO RE Sac CEC Uae ecco aCe oS Lecce model understand that "it" refers to "the animal" and not "the street.” It does this by creating a matrix of other word. This allows the model to build a deeply contextual understanding of syntax and semantics.POSITIONAL ENCODING Since Transformers process all words in a sentence at the same time, they inherently lack a sense of order. To a oS a a eC eee CUR OR ae ee cl crated introduced Positional Encodings. These are specific mathematical vectors added to the word embeddings that eet e aCe te CR Cee oan Cnt an este tet tarts restoring the critical grammatical information that was lost by removing the recurrent loop.12155 =)) 9) em a fe) VN Ba \ fete) ot} REPRESENTATIONS Ce OCC eC Transformers) changed the landscape by focusing on “bidirectionality.” Previous models read text from left-to-right (ike humans do). BERT, however, reads the entire text at once, os ae ee eRe aS oma PULA Eonar Ceuta) train this, BERT uses a "Cloze" task, where random words are ER Ree enemy ct eT Lee ere BERT to excel at understandingGPT: GENERATIVE PRE-TRAINED TRANSFORMER High Quality Samples Mode Coverage Diversity Fast Sampling While BERT focused on understanding, Ce eee Ut) tee eB reco COS ot eS itd Can OMe MICE Ey cisco eC a een Re Om Rot Ree Re RR ay the internet, GPT learned to generate Prost a eC cc? strength lies in its ability to continue anyTHE POWER OF TRANSFER LEARNING The success of models like BERT and GPT popularized Transfer Learning in NLP. In the past, every model had to be trained from scratch for a specific task (like sentiment analysis). Now, we take a massive model "pre-trained" on the general internet (which has learned the general rules of English) and "fine-tune" it on a small dataset for a specific job. This is analogous to teaching a human who already knows how to read to become a doctor. You drastically reduced the amount of labeled data needed to build world-class AI applications.ZERO-SHOT LEARNING One of the most surprising emergent properties of large language models is Zero-Shot Learning. This is the ability of a model to perform a task it was never explicitly trained to do. For example, you can ask GPT to “Translate this sentence to Spanish,” and it will do it, even if it was never trained specifically as a translation Pree RR eRe Sn eC amet eRe Re CeCe cece Seed training data. This suggests that at a certain scale, quantity [PeeeeLegi@ieeey transforms into quality, and simple next-word prediction evolves into general reasoning capabilities.TOKENIZATION: BYTE PAIR ENCODING Neural models do not read words exactly as humans do; they read “tokens.” A token can be a whole word, part of a word, or even a single character. Modern models use Byte Pair Encoding (BPE), an algorithm that breaks down text into the most efficient sub-word units. Common words like "the" remain single tokens, while rare SS ato ano Us O mL ee M aeeeEOROn CG eran the efficiency of whole-word vocabularies and the flexibility of character-based models.THE PROBLEM OF HALLUCINATION Dee ee Coa eo Pen tsneU CMs eee cm ome tee scr stance Cette to generate plausible-sounding text, they prioritize fluency over factuality. If a model does not know the answer, it will often. Cocca ee aoe eee ete eee eC a meee ood “what words usually go together.” This makes them dangerous eC ee ec oa CCST ees ot legal advice, without strict oversight and fact-checking mechanisms.BIAS AND ETHICAL CONCERNS Language models are trained on the internet, and the internet is a mirror of humanity—containing both our ee ee oem ene ea en eo en CCR eel Sree eee UCU a Se CC SS a ee ea Peet eRe ew Ce Teeter Ron GORA hee CCR tet eo Te eee coo oe CUA model [XPPEe Lr Pigseceed) toward safer and more unbiased responses, ensuring the AI serves all users eorieiaTHE SCALING LAWS eee EECCA TS eRe tec Che econ een ey Pot a ent ee Cte cen ee Rane et eeicanh simply making the model bigger and giving it more data consistently improves performance. This realization sparked the current arms race in Al, leading to models with hundreds of billions of parameters. However, we are to look for more efficient architectures rather than just blindly scaling up indefinitely.MULTIMODAL MODELS ‘The latest frontier in neural modeling is Multimodality. Language does not exist in a vacuum; it describes a Son ee CTC On Se Ree ene CS te een ae cet eects simultaneously. This allows an Al to understand the caption of an image, describe a video clip, or answer Cre OR haere Seer te eRe cae ECC eee cee td jor cneandy eee e ITE creates the word "sunset" is associated with evening”; it knows what a sunset actually looks like.FUTURE DIRECTIONS Mma) Vn Natural LUT Te bait] Rn Roa rece cored Reo comer ace Ree ccs te eR Peseta we Ce concert Cc ae Uo RCO MCAT} SOR e ae ee C enn hg oN mas Cog COS ee aCe tse Cree tee Sens toes Cee RS meee Pre en ra eka ressCONCLUSION Se Oe ea ee a eas Or te eat et tO ee eee a By mimicking the neural structure of the brain and leveraging the statistical properties of massive datasets, we Pe ena ec CCC Ron eae ORS Rooter e Re ices ee aU eee RCC mac Mtg ee tte ech Se aC CC CS co ee taeaeed) Suegests a future where the barrier between human and machine communication occa esQUESTIONS & ANSWERS Ria eon coger Prepared by Sir ArshadIMAGE SOURCESIMAGE SOURCES