Module 4+: Dynamic Programming, + Dynamic pHOgramming. : Dugnamic programming CDP) in senporcoment Leaxning & a model bosto , planning Suented frammwuorh, aed to Compute optimal policies when the enuronmont 48 anown and can Le modilled aha Makkoy decision process (MDP? “lt solues complex decision making, problems by busaking, trem down into simpler , evertapping and Leveraging the Ullman equations +b gind optimal policies "The agent thinks way ahead and choose tre Lest Qctions — Principles (1) Optimal substeuctune + A paoblem can be broken into smathn subpsoblems anc the solution to tno ounall problem depends on we otutions to Me bubproblems Eg Shortest poth in googie maps Finding the shoukess route from Mumbai to Debhi involves qinding the Shortest Moutes Letucen intounedia cites (2) Overlapping, subpHoblems The Some Bubproblems appear muttiplr dimes , 22 DP toss Solutions in a toble t auoid. sedunoant Computation—* Aduiontages ; *Guanontted optimal solution + Mathematical prucision, —> disadwantoges "Requires Gull model (naxe in ruck world.) —» Bellman equation : DP sulies entirely on the wecunsive Structune Of the Vou functions (1 Beldman equotion fox the State value function vs) ~ E[RG,a) t y V"(s')] Expanding, Hu, equation With transition pawbabi lity. vrlsd= 3 m(als) Z P(s'l6,2) [ Rls, a) trv" 6o)]who VT™s). value function of state $ ander poticy. P(e'|s,0.). Transition paobobility trom state $ é while tohing action a R(s,a): Rewand obtained ater toking action a ins 1 Discount factow for contratbing the importance Op futuste senards Tals) : Probabi lity vi taking action a in Sta 3 und pati — Optime) Stade valuL function V¥ Cs) 2 mote & pC4 6,4) (R60) ty V*s)] (2) Bellman. equation fox action Valuo function (9- function) g™(s,a) = E[R(8,a) t y v™s?] Expanding it using transition pxpbabilities Ws 9a) = Pfs! R6&, Yat) a™re! pt 9" (3,0) A (s'|¢ a)| Gat y X 1(a'[s!) 9"C#,a')] —> Optimal Action value Function Ot (sa): 5. P(e 18,0) [R(sla) ty mara! 9* (5,0) a* Potions ewaluation Poticy wvobuation is te PMOCLES Of deteumining the valu gunckien V"(S) fox a given peti, 1 +The value function sepuesents the erpocter, cumulative wunond tne agent will secetue if it follows the poticy 1 trom sate S$ —+ the bellman eqpation gor patioy evaluotion vis) = R(6) + ¥ > p(sils,m(s)) v™ (8) How v™(s) updates Atexately auntil it connrges tp tnd true value function for tro policy, —> Steps C1) Initialise v(s) =O forall te states (2) Rogen Update each stadt using, beleman equation (2a) Stop when sre eqpation Wiis te thushold Eg (si + [2+ Gok Action : Move aluays on suoht Rowotds : Sie 92=0 g2 —* goad = 10 Discount factor : y =0-4 step 4: Initialization v(si)=0 , VG2)=0, V( Goal) =0Stop 2: Update visi) = Rt y-v($2) =o+04%0 v(s1) =o V(S2) = Rt Y- VCgoat) =(0t 04XO V($2) = lO Step 3: Update v(sl) = R+ ¥-VCs2) = OF 0-9X10 v(sl) = 4 v(62) = Rt ¥ VCgoae) = 1040-71X0 V($2)= [0 rep +: Final Intexpretatron State Valur $I 4 $2 lo * Potion impxovement - Poti impsouemont is the Shep how the ogent spy a gah pling ating Ye ilu fran computed ousting tre policy evaluation ‘eT Bb a pabioy 5 its value function is VT) fon (oy i constructs a new poli re fs puedy aut fespeck 4D V™ ‘¢* Poticy, Impaouement cates a purdy potion » it pick He action that Looks Lust based on the ovyunt voluo estimates v™s) > VS) Eg ssi] = [2] Gow Y= 0-4 Rewaud : Moue to goat =o Othons mou = O Acgume poticy aaluation susults ag State Value $1 4 $2 5 Step 1: At SI Action Right > 82 GQ (S!, mignt) = Rt y-V(62) = OF04xXS 8 (SI, night) = 4-5 Action Left —» Grays (Assume 0) 9 (S81, Yt) = Rt y- VCsrays) =OLta Dea XO) SGI, YE) =o > Atusoys choose fuight fuom StStep 2: At 32 Action eye > 81: g (52, Ut) <0 +04 4-86 Action right + goat : 9 (82, sught) = lotO=ID —> Atwoys choose tugnt from S2 State «= Archon SI Rigi $2 Rights + Pabicy Henotion, + Poticy- iteration & an iterate (es Of Ampaouind, “It abteunates Lutween two steps - Step 1: Policy evaluation + Compute we volud op cach state tunden the cupvunt: potiog . vi ‘This is bolued iterate * Gives Row good wu consent poticyy 1 Step 2: Policy émprouement “It updabs tre policy in 2 gusedy manner Updates VT» y™ “This chooses the Lest action using tre cwoluated valueseg [Sc] e] soot Action: Left, wight Rawands : Reaching goat = +10 Othwuuise = O Discount factor - Y=0% Btep 1: Pati cup Cwoluaton Initotlyy , Sate Value s! Oo $2 0 — Pobiey dmnprovement, At Si, Right > 0 +0-9X0=0 Lept D cnosse always sgt jon 6! At $2, Lot + OF0-9KD=0 Ri (0 choose always Hight from &2 Updoded tr State Acton 3 RaeStep 2: Policy Waluation Prom S! , V(S1) = 0+0-%X10 vesiye From 62, v(62)= (OF 0-4X0 Vv(s2)= IO —Poticy Improvement At 61 Right =4 . hee eo Aways chosee tne ought At 62 Right = 10 Always choose the mi i ™ — optimal polioy State Achorn él Right $2 Right —> optimal Value function Sto = ACLLON S| ot $2 (o* Valu Iteration “Value iteration combines both the policy evaluation and the polidy impunvermont steps into a Single “It ieorortively updates tye value dunction using the bellman opmality egpation “Value iterotien often conven in toms o iterations shan potig itemation Anough each dtouation can Ye computationally Aoauy Eq [er] [2] Goat Action: Left, Hight Rowands : Reaching goak = tlo Otroumise = O Discount factor : Y=0% \nibiokigotean : Vp (61) =O, Yo(S2)=D » VCgoak)=0 Step 4: At $1, Right + O+0-9X0=0 Let —~ OF b-4AxD=D Vv, C81) =0 At 82, Right — |D + 0-4 x0=ID het - 0+0-9X0=0 ViC82) =10 Catooys chooee suight)Step 2: AL 81, Rught kept O+ O-4XID = 4 0 Vo(st) = % (Always chonse Hight) At 62, Right = (0+ 0-4 xd =|0 ht = 0 V2(62) = (0 CNo change) Hence , conuergence seached —» optimal Value function Store Value 81 o $2 10 —> Optimal state podicy state Action 8 2 ai * Applications of Dynanuc pidysamming (1) Rebotice A sey driving can oleciding. the best seute Pxoblem : The can must chopse the bapest and fastest path while considouing. tradpie signals ond Had ConditionsOP sotution : Value itenation fulps to calculate te Rust policy, based on past experiences (2) Healihicawt Optimal dua, dlasage planning Problem . A doctou must decide the optimal doug Onsage fox 0. diabetic patient +0 Maintal blood pols DP soutien : Potioy itmation 13 used tp adfust oluug, olosage achedutes for the Lest possible outcome # Appavaches for dynamic programming, () Top- olewn dynanue ae ‘In #us appeoach the agent solute te main psiob Lm gizst and thon sxecunbis Solkyes its subpaoblems »Stoving Heir susulee awptol supeoted compuiation “This method is called memouization > Working (1) Stouts with the O7iginal problem (2) Bsa it intd subproblems using secussion Ce) Stone tre sesitlts of each aubproblems in a table Consay (mop) (4) Rouke tre etoued ruautts when the tame subproblems Cow againSe aay - Bottom up dynamic puogramming soles Ale tre ematlor ‘uth pubes inst and un uses their seesults to huild the solution for ro Lasegen, problem “This method. is called a6 tabulation —> Working wn Ident py +ne bose cases : (2) 8olue subpuoblems iencootiuoly using Loops (2) Stove susulte in a tabte (4) Build up to tho final solution + Gereiabised policy iterortion CGP!) *GPl we a K that olescuibes how poticyy evaluation and poliup improvement interact and Continuously. Ampuove each oth to Hach an optimal abludion “Instead of shri Asponatin wakuation ond eee ish in peti Lteration , CrP) allows xhem to Wun eirmltoneeus Ly and pastiodly “Thy policy, waluation and ty policy impxoveront Sus tog HOM 7 4 ce each Bthiy and gradually conmbrge 4D optimal policy (r*) — Working [Evadiunte} limpuoue |... + Evaluation pushis values zownnds corociness + lmperovemn pushes policy towond optimality Eventuolly Lot stobilizesEq S| + 32 > goat Step 1. + Stout with sande poticy * Valuus ane inaccurate Step 2: * Do one update of the value * Immediately, impuoue policy Step 3: “Updake value again using, new pobicy *Imptoue the policy, again, Step 4: Ropoat Results « > Values Lucoms moxo acamate * Potion becoms optimal * Asynchronous dynamic PxOyrOMming, “Asynchronous dynamic proyramming a vassant Of DP whee the state values ose updated one at a time ot in subsets instead op updating. aul states sirurttanoousLy, and effectively Loaunung outcomes “In édondand Counchwensus) OP alt tre States ame apdared togetnen using Old values , bud in the asynchionous DP it updates One state at a ime and immediately use tre updated value fou the fudwu Uupdodtes—> Compenunts a Mubrip le agents “These oo ue indeptndent Leas expiouing diferent posts oF the envinonment Simultaneously each with its Leasing puocess Auk contributing, + he cumulative seuraxds Q) Shosud paxametens A common set Of poxamoters updated dy, the asynchronously. (3) Ranaller tuuade Asunchtonous methods Lwetage parallel peocessing, , ptsaeing id time and ie psrocessing eee (42 Siynchnonization mechanism Enews parameters and Rnowledge faom agents One Ofeckivelly aggregated and updated —> Methods OP implementation CD Pautial Seep Instead of updating Ale te states in cach dtecation the algoaitim Selects @ Aubset OF states gor upaoding- C2) Selectine upolotes States ase most Likely 4p wad +0 improv ements Can we prioritized fou upolatesaad Types CD In phaco value [tewation + Updates State one by one » Utes Latest updated values smmediately 2) Roal time olynamic - Updates ondy. States visited using, intewaction C3) Prionitized “Update important stotes fisst * Priovity. is based on Losgen value changes and Figh impact tates