0% found this document useful (0 votes)
12 views15 pages

Week 8 - Variational Inference - AC

The document discusses variational inference as a method for posterior inference in the context of artificial intelligence. It highlights the challenge of inferring the posterior distribution of an unknown latent variable Z given an observed variable X, which is often intractable. The goal is to find a distribution that approximates the true posterior distribution closely.

Uploaded by

Erjon Brucaj
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
0% found this document useful (0 votes)
12 views15 pages

Week 8 - Variational Inference - AC

The document discusses variational inference as a method for posterior inference in the context of artificial intelligence. It highlights the challenge of inferring the posterior distribution of an unknown latent variable Z given an observed variable X, which is often intractable. The goal is to find a distribution that approximates the true posterior distribution closely.

Uploaded by

Erjon Brucaj
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd

ECE 57000 – Artificial Intelligence

Variational Inference

Murat Kocaoglu
sha1_base64="tFD/VhBS/RMB79tt72IuNNNWPzg=">AAAJDnicjVbRjhs1FJ0WSrOh0BYeeTEkK5UqG5K2Krys1NXy0AeQwqrbVspElcfjJFY8tmV7ukmt+Qb6WH6EN8Qrv8B38ANcT5zNzCRBTBTl5txzj32vr+1JFGfGDgZ/37j50ce3PrndOmp/euezz+/eu//FSyNzTeglkVzq1wk2lDNBLy2znL5WmuIs4fRVsjj3/ldvqTZMihd2pegkwzPBpoxgC9ClevDuW/TmXmfQH5QP2jWGwehE4Rm9uX/7nziVJM+osIRjY8bDgbITh7VlhNOiHeeGKkwWeEbHYAqcUTNx5WwLdAxIiqZSw1dYVKLVCIczY1ZZAswM27lp+jx4yGfn2bWrHQt6RWSWYZG62M7pc8xnUjPPcTHWOGHEbaGiXRfbOBgp9jvqAePcTn+YOCZUbqkg6zSnOUdWIl94lDJNieUrMDABAUYQmcMsiIXlqSmlb5kyoWbLddFgKEhAagoprjh1imMmijLDgLvwW7gXwRgbGA8WeVKjKS2VNMw7inEAJ25UQWt0TqF+FeJP5f8ahUgtOcd6VaGdX2ONiad0ysSecSr4VuXHA2TouDxTDfLZFmyMCQbWi7pEwLbhF4FU7xomUqoQtIuiWsVflx9vFnUaYcX48cR1hihEdB6hmNOpjWHvWdR5HGs2m9t+I0xU4oS0QbsyzGGdWktatnhXb1KTJwRvilHBs5xbpuVVsXdTtZuo36CmoQBEKyU3DfbSgEUbCSpFCue68ciw7snovIj3snBuZYZV4c7A+PlsdM2q00yZO9Z++u5hWaMK0uD6ogVqfG37mIdNJrj8dyOZJXLpTk5OikBusAnT/oRbk7tbdlfutIQnbuhyy9xPXE9BVoffN9Pq8AcVLWZ8rbfl7GRCqtWU/11NcrCaO8l4b025ViN0sHG9xVmi4chwZajpQadpttx1mzlW1PQS0JhpmYsUWvH4ACmlcDqVV53pBVmCOQl2P6MW96bM9jaHHxMzL+aVQM+1j4+UZMJ+1y9PEnSK3HoFeijV+KqHmBBUI0joFO7LJyQDMc57QqYU7jUBF4E1pw6ulmN/Jm1Ej7zG/5fcVdutSW3+1QrPNFZzRpbN5V/AbnTdhd+VqLGGF8V4OIFN63c6FMtdFJ1hF+LhNWHYfCnYNV4+6g+f9ge/POk8expeGFrRV9E30YNoGH0fPYueR6PoMiIRi36NPkS/td63fm/90fpzTb15I8R8GdWe1l//AsxuT1E=</latexit>
<latexit
Bayes rule:
sha1_base64="14haB/8T2vjCfUoR2XLIJcBzZTE=">AAAJVXicjVZdb9s2FFW7ro29j6br41642QGSwvHstuj6EqBB9lAMG+AFTVvAMgKKom3CFEmQVGOb1X/ar9mwp+0/7HnALmU5lmR7mAzD1+eee8h7eUkqUpwZ2+v9cefuJ/c+vf/goNH87PMvvnx4+Oirt0ammtArIrnU7yNsKGeCXllmOX2vNMVJxOm7aHbh/e8+UG2YFG/sQtFRgieCjRnBFqDrwx/V8fLj/ASdoXCsMXHqeP5xeQLgSebtk2ynJ2TCXi/RBkLxMrs+bPW6vfxB20a/MFpB8QyuHz34O4wlSRMqLOHYmGG/p+zIYW0Z4TRrhqmhCpMZntAhmAIn1IxcnnSGjgCJ0Vhq+AqLcrQc4XBizCKJgJlgOzV1nwf3+ew0uXU1Q0FviEwSLGIX2il9jflEauY5LsQaR4y4DZQ1q2JrByPZbkc1YJja8cuRY0KllgqySnOccmQl8uuHYqYpsXwBBiYgwAgiU5gFsbDKFaX4A1OmqNl8VTQYChKQmkKKC06d4piJLM+wwF3xm7k3hTE0MB70yqhCU1oqaZh3ZMMCHLlBCa3QOYX6lYg/5f8rFCK15BzrRYl2cYvVJh7TMRM7xinhG5Uf9pCh49JE1cjnG7A2JhhYz6oSBbYJvyxI1a5hIqYKQbsoqlX4Tf7xZlalEZYNn41cq4+KiNZTFHI6tiFsYYtaz0LNJlPbrYWJUpyQttAuDbNfp9KSls2W1SY1aUTwuhglPEm5ZVreZDs3VbOO+g1qagpAtFJyU2PPDVi0lqBSJHOuHQ4Ma58OLrJwJwunViZYZe4cjJ/PB7esKs3kuWPtp++e5DUqITWuL1pBDW9tH/OkzgSX/64lk0jO3enpaVaQa2zCtD/hVuT2ht2WWy3hiWu63DB3E1dTkOXhd820PPxeRYsZX+ltOFuZkHI15X9Xk+yt5lYy3ltRrtQI7W1cb3EWaTgyXB5qOtBpms233WaKFTWdCDQmWqYihlY82kOKKZxO+Y1pOoUswZwUdjehFnfGzHbWhx8TEy/mlUDPNY8aSsKV+V03P0ngSnWrFeigWOObDmJCUI0goTO4L5+TBMQ47wgZU7jXBFwE1pw5uFqO/Jm0Fm14jf8vua22XZPK/MsVnmispozM68s/g93o2jO/K1FtDS+zYX8Em9bvdCiWu8xa/TbEw2tCv/5SsG28fdrtv+j2fnneevWieGE4CL4Ovg2Og37wffAqeB0MgquABL8GvwV/Bn8d/H7wT+Ne4/6KevdOEfM4qDyNh/8CKO5n6g==</latexit>
<latexit
p(z|x) =
p(x)
p(x|z)p(z)
Posterior Inference

sha1_base64="Y6AZgqLID/iRePQRILV6U7UaK4I=">AAAJEHicjVbNjts2EFbSn3jdv6Q59sLWXiANvK6dBEkvC2SxPeTQAu4imw1gGQFF0TZhiiRIKmuH1UO0x/ZFeit67Rv0OfoCHcr0WpLtojIMj7/55iNnOCSVKM6MHQz+vnX7vfc/+PBO66j90ceffPrZ3XufvzIy14ReEsmlfp1gQzkT9NIyy+lrpSnOEk6vksW591+9pdowKV7alaKTDM8EmzKCLUBX6sHyp3dfozd3O4P+oHzQrjEMRicKz+jNvTv/xKkkeUaFJRwbMx4OlJ04rC0jnBbtODdUYbLAMzoGU+CMmokr51ugY0BSNJUavsKiEq1GOJwZs8oSYGbYzk3T58FDPjvPblztWNBrIrMMi9TFdk5fYD6TmnmOi7HGCSNuCxXtutjGwUix31EPGOd2+u3EMaFySwVZpznNObIS+dKjlGlKLF+BgQkIMILIHGZBLCxQTSl9y5QJNVuuiwZDQQJSU0hxxalTHDNRlBkG3IXfwr0MxtjAeLDMkxpNaamkYd5RjAM4caMKWqNzCvWrEL8v/9coRGrJOdarCu38BmtMPKVTJvaMU8G3Kt8dIEPH5ZlqkM+2YGNMMLBe1CUCtg2/CKR61zCRUoWgXRTVKv6y/HizqNMIK8aPJ64zRCGi8wjFnE5tDLvPos7jWLPZ3PYbYaISJ6QN2pVhDuvUWtKyxbt6k5o8IXhTjAqe5dwyLa+LvZuq3UT9BjUNBSBaKblpsJcGLNpIUClSONeNR4Z1T0bnRbyXhXMrM6wKdwbGD2ejG1adZsrcsfbTdw/LGlWQBtcXLVDjG9vHPGwyweW/G8kskUt3cnJSBHKDTZj2J9ya3N2yu3KnJTxxQ5db5n7iegqyOvy+mVaHP6hoMeNrvS1nJxNSrab872qSg9XcScZ7a8q1GqGDjestzhINR4YrQ00POk2z5a7bzLGippeAxkzLXKTQiscHSCmF06m87EwvyBLMSbD7GbW4N2W2tzn8mJh5Ma8Eeq59fKQkE/abfnmSoFPk1ivQQ6nG1z3EhKAaQUKncF8+IRmIcd4TMqVwrwm4CKw5dXC1HPszaSN65DX+v+Su2m5NavOvVnimsZozsmwu/wJ2o+su/K5EjTW8KMbDCWxav9OhWO6i6Ay7EA+vCcPmS8Gu8epRf/i0P/jxSef50/DC0Iq+iL6KHkTD6Fn0PHoRjaLLiESL6Ofo1+i31i+t31t/tP5cU2/fCjH3o9rT+utfZ89QWQ==</latexit>
<latexit
R
z
Goal: Infer the posterior distribution of Z.

p(x|z)p(z)
p(x|z)p(z)dz
Unknown latent variable Z, known observed variable X.

Usually intractable!

finding a distribution that is close in some distance to true posterior.

sha1_base64="mJnZp4KTMTKKfCmkn+5CBNkn2C4=">AAAJD3icjVbdjhs1FJ6Wn2bDT1u45MaQrFSqbEjaqnCzUlfLRS9ACqvutlImqjweJzHx2Jbt6SY18w5wCS/CHeKWR+A5eAGOJ85mZpIgJopy8p3vfPY5PrYnUZwZOxj8fev2O+++9/6d1lH7gw8/+vjuvfufXBmZa0IvieRSv0qwoZwJemmZ5fSV0hRnCacvk8W59798Q7VhUrywK0UnGZ4JNmUEW4Cu1IO3Py2/fH2vM+gPygftGsNgdKLwjF7fv/NPnEqSZ1RYwrEx4+FA2YnD2jLCadGOc0MVJgs8o2MwBc6ombhyugU6BiRFU6nhKywq0WqEw5kxqywBZobt3DR9Hjzks/PsxtWOBb0mMsuwSF1s5/Q55jOpmee4GGucMOK2UNGui20cjBT7HfWAcW6n30wcEyq3VJB1mtOcIyuRrzxKmabE8hUYmIAAI4jMYRbEwvrUlNI3TJlQs+W6aDAUJCA1hRRXnDrFMRNFmWHAXfgt3ItgjA2MB6s8qdGUlkoa5h3FOIATN6qgNTqnUL8K8bvyf41CpJacY72q0M5vsMbEUzplYs84FXyr8u0BMnRcnqkG+WwLNsYEA+tFXSJg2/CLQKp3DRMpVQjaRVGt4s/LjzeLOo2wYvx44jpDFCI6j1DM6dTGsPks6jyONZvNbb8RJipxQtqgXRnmsE6tJS1bvK03qckTgjfFqOBZzi3T8rrYu6naTdRvUNNQAKKVkpsGe2nAoo0ElSKFc914ZFj3ZHRexHtZOLcyw6pwZ2B8fza6YdVppswdaz9997CsUQVpcH3RAjW+sX3MwyYTXP67kcwSuXQnJydFIDfYhGl/wq3J3S27K3dawhM3dLll7ieupyCrw++baXX4g4oWM77W23J2MiHVasr/riY5WM2dZLy3plyrETrYuN7iLNFwZLgy1PSg0zRb7rrNHCtqeglozLTMRQqteHyAlFI4ncq7zvSCLMGcBLufUYt7U2Z7m8OPiZkX80qg59rHR0oyYb/qlycJOkVuvQI9lGp83UNMCKoRJHQK9+UTkoEY5z0hUwr3moCLwJpTB1fLsT+TNqJHXuP/S+6q7dakNv9qhWcaqzkjy+byL2A3uu7C70rUWMOLYjycwKb1Ox2K5S6KzrAL8fCaMGy+FOwaV4/6w6f9wQ9POs+ehheGVvRZ9EX0IBpGX0fPoufRKLqMSPRj9HP0a/Rb65fW760/Wn+uqbdvhZhPo9rT+utfJMVQLw==</latexit>
<latexit
Variational inference converts this problem into an optimization problem of
Given prior p(z) and data likelihood p(x|z) compute the posterior of latent p(z|x)
sha1_base64="Am0fTDXpTKQjR0ZyWPcDuoaJMsQ=">AAAJSHicjVbNbhs3EN64P5HVP6c99sJWMmAHsiolQdqLgRjOIUBTQDHiJIBWECguJRHikjTJjSXT+zR9mvbYnvoWBXooeutwtbJ2V1LRFRY7mvnmI2c45HCkODO20/nj3t4HH3708f3afv2TTz/7/IuDB1++MTLRhF4SyaV+N8KGcibopWWW03dKUxyPOH07mp17+9v3VBsmxWu7UHQQ44lgY0awBdXw4PQKnaIQ60kYMzF0VyETKIyxnRLM3as0Rc+H7seX6dHV0c3x7a0autBOqcXp0c3t/Ph4eNDotDvZgzaFbi40gvzpDR/c/yuMJEliKizh2Jh+t6PswGFtGeE0rYeJoQqTGZ7QPogCx9QMXBZnig5BE6Gx1PAKizJt0cPh2JhFPAKkD8FUbV65y2an8Z2pHgp6TWQcYxFlAb/AfCI18xgHycIjRtxaldbLZCsDI+l2Q9mhn9jxDwPHhEosFWQZ5jjhyErklwxFTFNi+QIETICAEUSmMAtiYWFLTNF7pkyes/kyaTAUBCA1hRAXnDrFMRNpFmGud/k3da9zoW9gPCiPQQmmtFTSMG9I+7ly4HoFbQnOKeSvAHyZ/S9BiNSSc6wXBdj5na4y8YiOmdgyTkG/Znm+AwwVl8SqAj5bKytjgoD1rEyR69buFzmoXDVMRFQhKBdFtQq/yX5eTMswwtL+44FrdFHu0XiEQk7HNoRda1HjcajZZGrbFTdR8BPS5tyFYXbzlErSstlNuUhNMiJ4lYyCPk64ZVpep1s3Vb2q9RvUVBgAaKXkpoKeG5BoJUClSOpcM+wZ1jzpnafhVhROrIyxSt0ZCD+d9e5QZZjJYsfaT989zHJU0FSwPmk5NLyTvc/DKhJM/l1RxiM5dycnJ2kOrqAJ0/6EW4Kba3RTbpSEB67gco3cDlxOQRaH3zbT4vA7GS1mfMm3xmxEQorZlP+dTbIzmxvBeGuJuZQjtLNwvcTZSMOR4TJX04JK02y+aTZTrKhpjYBjomUiIijFwx2giMLplDVJ08ppoSWSXG7H0ARbY2Zbq8OPiYkn80zA5+qH+0oyYb9rZycJNFm3XIEWijS+biEmBNUIAjqFfvmExEDGeUvIiEJfE9AIrDl10FoO/Zm0It33HP+fcpNtMyel+RczPNFYTRmZV5d/BrvRNWd+V6LKGl6k/e4ANu3q/nCRNrpN8IdrQrd6KdgU3jxqd5+2O6+eNJ49zS8MteDr4NvgKOgG3wfPghdBL7gMSPBz8EvwW/B77dfan7W/a/8soXv3cp+vgtKzv/cvycNl5Q==</latexit>
<latexit
q2Q

Rewrite the loss:


q = arg min DKL (q(z)||p✓ (z|x))
Variational Inference

Usually, q is searched within a family of distributions characterized by some parameter, say

sha1_base64="qET42eNBl6RYTZsxqRFHMpQuEPg=">AAAJDXicjVbBjts2EFXSpvE6aZs0x17Y2gsUgde1kyDtZYEstoccWsBZZJMAlhFQFG0TpkiCpLJ2CP1Ce2x+JLei135Dv6M/0KFMryXZLirD8PjNm0fOcEgqUZwZOxj8fePmJ5/e+ux266h95+7nX3x57/5Xr4zMNaGXRHKp3yTYUM4EvbTMcvpGaYqzhNPXyeLc+1+/o9owKV7alaKTDM8EmzKCrYdiZdjbe51Bf1A+aNcYBqMThWf09v7tf+JUkjyjwhKOjRkPB8pOHNaWEU6LdpwbqjBZ4BkdgylwRs3ElZMt0DEgKZpKDV9hUYlWIxzOjFllCTAzbOem6fPgIZ+dZ9eudizoFZFZhkXqYjunzzGfSc08x8VY44QRt4WKdl1s42Ck2O+oB4xzO/1x4phQuaWCrNOc5hxZiXzdUco0JZavwMAEBBhBZA6zIBZWp6aUvmPKhJot10WDoSABqSmkuOLUKY6ZKMoMA+7Cb+FeBmNsYDxY40mNprRU0jDvKMYBnLhRBa3ROYX6VYg/l/9rFCK15BzrVYV2fo01Jp7SKRN7xqngW5WfDpCh4/JMNchnW7AxJhhYL+oSAduGXwRSvWuYSKlC0C6KahV/U368WdRphBXjxxPXGaIQ0XmEYk6nNoatZ1HncazZbG77jTBRiRPSBu3KMId1ai1p2eJ9vUlNnhC8KUYFz3JumZZXxd5N1W6ifoOahgIQrZTcNNhLAxZtJKgUKZzrxiPDuiej8yLey8K5lRlWhTsD45ez0TWrTjNl7lj76buHZY0qSIPrixao8bXtYx42meDy341klsilOzk5KQK5wSZM+xNuTe5u2V250xKeuKHLLXM/cT0FWR1+30yrwx9UtJjxtd6Ws5MJqVZT/nc1ycFq7iTjvTXlWo3Qwcb1FmeJhiPDlaGmB52m2XLXbeZYUdNLQGOmZS5SaMXjA6SUwulU3nSmF2QJ5iTY/Yxa3Jsy29scfkzMvJhXAj3XPj5Skgn7fb88SdApcusV6KFU46seYkJQjSChU7gvn5AMxDjvCZlSuNcEXATWnDq4Wo79mbQRPfIa/19yV223JrX5Vys801jNGVk2l38Bu9F1F35XosYaXhTj4QQ2rd/pUCx3UXSGXYiH14Rh86Vg13j1qD982h+8eNJ59jS8MLSir6Nvo++iYfRD9Cx6Ho2iy4hE8+jX6PfoQ+u31sfWH60/19SbN0LMg6j2tP76FzmPT5Q=</latexit>
<latexit
Adds the same constant for any q.
where
Thus, we have:

sha1_base64="sxJm87CAy3vVIFG+A7FGBLRp+W4=">AAAJXnicjVZBjxo3FJ4kbbLQptm0l0q9uIWV2ISlkERpLytltTlEairRbTaJxCBkPAYsPLZje7LseueH9af01B7bc/9An4dhmRmg6iDE43vf++z3/GzPWHFmbLf7+63bdz759O69vVr9s8/vf/Fg/+GXb41MNKHnRHKp34+xoZwJem6Z5fS90hTHY07fjeen3v/uI9WGSfHGXio6jPFUsAkj2AI02v/15cj99DptfRiFyrDW1eH1tRq50M6oxWnr6npxeIiOURhjOyOYO2AufW1PBy96jEIupy01WuItCBjtN7qdbvagTaOXG40gf/qjh/f+CSNJkpgKSzg2ZtDrKjt0WFtGOE3rYWKowmSOp3QApsAxNUOXZZ+iA0AiNJEavsKiDC1GOBwbcxmPgenTMFWfB3f57Cy+cdVDQS+IjGMsoqxCrzCfSs08x4VY4zEjbg2l9bLYysFIut1RDhgkdvLj0DGhEksFWaY5STiyEvmFRBHTlFh+CQYmIMAIIjOYBbGw3CWl6CNTJq/ZYlk0GAoSkJpCipecOsUxE2mWYY67/Dd1b3JjYGA8aJphiaa0VNIw70gHOTh0/QJaonMK9SsQX2f/SxQiteQc68sC7fQGq0w8ohMmtoxTwNcqL3eQoeOSWFXIJ2uwMiYYWM/LEjm2Dj/LSeWuYSKiCkG7KKpV+G328WZaphGWDp4OXaOH8ojGE9hmdGJD2MsWNZ6Gmk1ntlMJE4U4IW2uXRhmt06pJS2bX5Wb1CRjglfFKOBxwi3T8iLduqnqVdRvUFNRAKKVkpsKe2HAopUElSKpc82wb1jzqH+ahltZOLEyxip1J2D8fNK/YZVpJssdaz999yirUQGpcH3Rcmp4Y/uYR1UmuPx3JRmP5cIdHR2lObnCJkz7E25Jbq7ZTbnREp64oss1cztxOQVZHH7bTIvD71S0mPGl3pqzkQkpVlP+dzXJzmpuJOO9JeVSjdDOxvUWZ2MNR4bLQk0bOk2zxabbzLCipj0GjamWiYigFQ92kCIKp1N2dZp2LgvXIsntTuxvxgmz7dXhx8TUi3kl0HP1g5qSTNjvO9lJAjerW65AG0UaX7QRE4JqBAkdw335jMQgxnlbyIjCvSbgIrDm2MHVcuDPpJVozWv8f8lNtc2alOZfrPBUYzVjZFFd/jnsRtec+12JKmt4lg56Q9i0q3eIs7TRa0I8vCb0qi8Fm8bbJ53e8073l2eNF8/zF4a94Jvgu6AV9IIfghfBq6AfnAck+C34I/gr+Hvvz9rd2v3agyX19q085qug9NS+/heDA2rT</latexit>
<latexit
Similarly, define
sha1_base64="DT1hPRalDPMYm8DraD1F2M9UVkA=">AAAJPXicjVZPjxs1FJ+2QLPhT7flyMWQrLRbJSHZVoXLSl0thx6KFFbdtlImWnk8TmLFYxvb000w8z34NHCEr8AH4Ia4cODK82SymZkkiImivPze7/3s9/xsT6Q4M7bf//3O3Xvvvf/B/cZB88OPPv7kweHDR6+NTDWhV0Ryqd9G2FDOBL2yzHL6VmmKk4jTN9H8wvvfvKPaMCle2aWi4wRPBZswgi1A14enYYLtjGDuXmbHoZ1RizuhMuzHxQkKp/R71A25nB6r65XveHFycn3Y6vf6+YO2jUFhtILiGV4/vP93GEuSJlRYwrExo0Ff2bHD2jLCadYMU0MVJnM8pSMwBU6oGbs8uQwdARKjidTwFRblaDnC4cSYZRIB06di6j4P7vPZWXLraoaC3hCZJFjEzmf7AvOp1MxzXIg1jhhxGyhrVsXWDkay3Y5qwCi1k6/HjgmVWirIKs1JypGVyK8TipmmxPIlGJiAACOIzGAWxMJqVpTid0yZomaLVdFgKEhAagopLjl1imMmsjzDAnfFb+ZeFcbIwHjQE+MKTWmppGHekY0KcOyGJbRC5xTqVyK+zP9XKERqyTnWyxLt4harTTymEyZ2jFPCNyrf7CFDx6WJqpHPN2BtTDCwnlclCmwTflmQql3DREwVgnZRVKvw8/zjzaxKIywbPRm71gAVEa1TFHI6sSFsVYtaT0LNpjPbq4WJUpyQttAuDbNfp9KSls1/qDapSSOC18Uo4UnKLdPyJtu5qZp11G9QU1MAopWSmxp7YcCitQSVIplz7XBoWLs7vMjCnSycWplglblzML49H96yqjST5461n757nNeohNS4vmgFNby1fczjOhNc/ruWTCK5cN1uNyvINTZh2p9wK3J7w27LrZbwxDVdbpi7iaspyPLwu2ZaHn6vosWMr/Q2nK1MSLma8r+rSfZWcysZ760oV2qE9jautziLNBwZLg81Heg0zRbbbjPDippOBBpTLVMRQyse7SHFFE6n/GY0nUIWrkZS2L3E344TZjvrw4+JqRfzSqDnmkcHSjJhv+zlJwk6Q261Ah0Ua3zTQUwIqhEkdAb35VOSgBjnHSFjCveagIvAmjMHV8uRP5PWogde4/9Lbqtt16Qy/3KFpxqrGSOL+vLPYTe69tzvSlRbw8tsNBjDpl2/R1xmrUEb4uE1YVB/Kdg2Xp/2Bs96/e+etp4/K14YGsFnwRfBcTAIvgqeBy+CYXAVkOCn4Ofg1+C3xi+NPxp/Nv5aUe/eKWI+DSpP459/AdJ0Ye0=</latexit>
<latexit
Since KL is always non-negative, we have
Variational Inference

, then we have
variational free energy free energy
L(✓, |x)   log(p✓ (x))

evidence lower bound data log likelihood


DKL (q (z)||p✓ (z|x)) = L(✓, |x) + log(p✓ (x))
Variational Inference
ELBO
Several ways to write the ELBO:

1.

1’,

2.

3.
sha1_base64="k7wwv8rNWHYZy1M0PhXjCx7R39g=">AAAJP3icjVbNjts2EFa2f173J5v22Atb28Am8Lp2EqS9LJDttkAOKeouskkAyzAoirYJUyRLUlm7rJ6gT9Nre+pj9Al6K3rtrUNZXkuyDVSG4fHMNx85w5mhIsWZsf3+n3eO3nr7nXffaxw33//gw4/untz7+KWRqSb0mkgu9esIG8qZoNeWWU5fK01xEnH6KlpcevurN1QbJsULu1J0nOCZYFNGsAXV5KQTWrq07tvnX3+fTUTI6Y8o5HJ2qk6XE/FzaOfU4vv3Jyetfq+fP2hXGBRCKyie4eRe4yiMJUkTKizh2JjRoK/s2GFtGeE0a4apoQqTBZ7REYgCJ9SMXR5PhjqgidFUavgKi3Jt2cPhxJhVEgEywXZu6javPGSz8+TW1AwFvSEySbCInY/1GeYzqZnHuBBrHDHitqqsWSXbGBjJ9huqDqPUTr8aOyZUaqkg6zCnKUdWIn80KGaaEstXIGACBIwgModdEAsHWGGK3zBlipwt10mDpSAAqSmEuOLUKY6ZyPIIC70rfjP3ohBGBtaDMhhXYEpLJQ3zhmxUKMduWNJW4JxC/krA5/n/CoRILTnHelWCXd7qahuP6ZSJPeuU9FuWbw6AoeLSRNXAF1tlbU0QsF5UKQrd1v2qAFWrhomYKgTloqhW4Wf5x4tZFUZYNno0dq0BKjxaD6HR6NSG0J0WtR6Fms3mtldzEyU/IW3BXVrmME+lJC1b/FQtUpNGBG+SUdInKbdMy5tsb1M161rfoKbGAEArJTc19NKARGsBKkUy59rh0LD22fAyC/eicGplglXmLkD47mJ4i6rCTB471n777kGeo5KmhvVJK6Dhrex9HtSRYPLfDWUSyaU7OzvLCnANTZj2E24Nbm/RbblTEh64gcstcj9wvQVZXn7fTsvLH2S0mPE13xazEwkpZ1PWsrk3p+RgTndC8tYKfyVT6GD5eomzSMPgcLmr6UK9abbsKmxhTgqzizNzrKjpRkA20zIVMWA6B0AxhWGV342mW/ATzEkh9xK4DrtTZrubWcjEzJN5JuBzzc6xkkzYL3r5YEHnyK0PpItijW+6iAlBNYLIzuH6fEwSIOO8K2RM4ZoTcC9Yc+7gpun4EbUhPfYc/59yl203J5X9l1M901jNGVnWq2EBzenaC9+kqHaYV9loMIYe9o0PyXJXWWvQBn94axjU3xF2hZcPe4Mnvf4Pj1tPnxTvD43g0+Dz4DQYBF8GT4NnwTC4DkjwS/Br8Fvwe+OPxl+Nvxv/rKFHdwqfT4LK0/j3P/PbYqc=</latexit>
<latexit
sha1_base64="3wv8g1l5HwiTbl8CC+izcwAgdxQ=">AAAJUHicjVZdjxs1FJ2Wjybho1t45MWQrLRU2ZC0VUFIK3W1ReoDoLDqtpUyUeTxOIkVj23Znm6yZn4RvwbegJ/BC29wPZlsZiYJYqIoN+eee+x7fW1PpDgztt///c7dd9597/17jWbrgw8/+vj+0YNPXhmZakKviORSv4mwoZwJemWZ5fSN0hQnEaevo8WF979+S7VhUry0K0XHCZ4JNmUEW4AmR9+pkzDBdk4wd8+z7s3EDb79Mfs5tHNq8ZfoLFRaxhOB1MnNRGxgdbKEPwB0C2By1O73+vmDdo1BYbSD4hlOHtz7K4wlSRMqLOHYmNGgr+zYYW0Z4TRrhamhCpMFntERmAIn1Ixdnm+GjgGJ0VRq+AqLcrQc4XBizCqJgOlzM3WfBw/57Dy5dbVCQa+JTBIsYudTfYH5TGrmOS7EGkeMuC2UtapiGwcj2X5HNWCU2uk3Y8eESi0VZJ3mNOXISuSXDsVMU2L5CgxMQIARROYwC2JhgStK8VumTFGz5bpoMBQkIDWFFFecOsUxE1meYYG74jdzLwtjZGA8aJNxhQYdoaRh3pGNCnDshiW0QucU6lcifp//r1CI1JJzrFcl2sUtVpt4TKdM7BmnhG9Vnh8gQ8eliaqRz7dgbUwwsF5UJQpsG35ZkKpdw0RMFYJ2UVSr8PP8482sSiMsGz0eu/YAFRHtRyjkdGpD2L0WtR+Hms3mtlcLE6U4IW2hXRrmsE6lJS1b3FSb1KQRwZtilPAk5ZZpeZ3t3VStOuo3qKkpANFKyU2NvTRg0VqCSpHMuU44NKxzOrzIwr0snFqZYJW5czB+OB/esqo0k+eOtZ++e5jXqITUuL5oBTW8tX3MwzoTXP67kUwiuXSnp6dZQa6xCdP+hFuTO1t2R+60hCdu6HLL3E9cT0GWh9830/LwBxUtZnytt+XsZELK1ZT/XU1ysJo7yXhvRblSI3Swcb3FWaThyHB5qOlCp2m23HWbOVbUdCPQmGmZihha8fgAKaZwOuWXpekWsnBXksLuJXD9dafMdjeHHxMzL+aVQM+1jptKMmG/6uUnCTpDbr0CXRRrfN1FTAiqESR0BvflE5KAGOddIWMK95qAi8CaMwdXy7E/kzaiTa/x/yV31XZrUpl/ucIzjdWckWV9+RewG11n4Xclqq3hZTYajGHTbl4sLrP2oAPx8JowqL8U7BqvHvUGT3v9n560nz0tXhgawWfBF8FJMAi+Dp4FL4JhcBWQ4Jfg1+CP4M/Gb42/G/8076ypd4vf4NOg8jRb/wLZFmg4</latexit>
<latexit
Data is x:

Repeat:
p(D, z1:N |✓) =

Intractable, use ELBO:


n
Y

ELBOn  log(p(xn |✓))

Run EM to maximize ELBO:


sha1_base64="AoyT4cHHVT0p+MAZ+t/LVBeQufs=">AAAJTXicjVbNjts2EFa2P/G6f5v22Atbe4FN4HXtbJD0skA2mwI5pKi7yCYBLGNBUbRNmCIZksraZfQgfZpe21PPfZDcgqJDWV5Lsg1UhuDRzDcfOcPhkJHizNhe759bex99/Mmntxv7zc8+/+LLrw7ufP3SyFQTekkkl/p1hA3lTNBLyyynr5WmOIk4fRXNzr391VuqDZPihV0oOkrwRLAxI9iC6urgJDRpciVCS+fW/fT8yS8ZfHD6BoVcTo7UUZhgOyWYu6fZu9BOqcV3714dtHrdXv6gTaFfCK2geAZXdxp7YSxJmlBhCcfGDPs9ZUcOa8sIp1kzTA1VmMzwhA5BFDihZuTy6DJ0CJoYjaWGV1iUa8seDifGLJIIkH6ypm7zyl02O01uTM1Q0GsikwSL2PlYn2E+kZp5jAuxxhEjbq3KmlWylYGRbLuh6jBM7fjHkWNCpZYKsgxznHJkJfILhWKmKbF8AQImQMAIIlOYBbGwnBWm+C1TpsjZfJk0GAoCkJpCiAtOneKYiSyPsNC74j9zLwphaGA8KIpRBaa0VNIwb8iGhXLkBiVtBc4p5K8EfJ5/VyBEask51osS7PxGV5t4TMdMbBmnpF+zPN0BhopLE1UDn62VtTFBwHpWpSh0a/eLAlStGiZiqhCUi6Jahd/lPy9mVRhh2fBk5Fp9VHi07sOGo2Mbwl61qHUSajaZ2m7NTZT8hLQFd2mY3TyVkrRs9lu1SE0aEbxKRkmfpNwyLa+zrZuqWdf6DWpqDAC0UnJTQ88NSLQWoFIkc64dDgxrHw/Os3ArCqdWJlhl7gyEn88GN6gqzOSxY+2n7+7lOSppaliftAIa3sje514dCSb/riiTSM7d8fFxVoBraMK073BLcHuNbsuNkvDAFVyukduByynI8vDbZloefiejxYwv+daYjUhIOZuyls2tOSU7c7oRkrdW+CuZQjvL10ucRRoah8tdTQfqTbN5R2ELfVKYTZyZYkVNJwKyiZapiAFzuAMUU2hW+UlpOgU/nIWkkLsJHIedMbOdVS9kYuLJPBPwuebhvpJM2B+6eWNBp8gtF6SDYo2vO4gJQTWCyE7h+HxAEiDjvCNkTOGYE3AuWHPq4KQ59C1qRbrvOf4/5SbbZk4q8y+neqKxmjIyr1fDDDana8/8JkW1xbzIhv0R7OHVxeEia/Xb4A+3hn79jrApvLzf7T/s9n590Hr8sLg/NIJvg++Do6AfPAoeB8+CQXAZkOD34I/gz+Cvxt+N940PjX+X0L1bhc83QeXZv/0fsBtn3w==</latexit>
<latexit
n
Variational EM

sha1_base64="Lg05fZvkcfSeVSfcojEbgu+aszM=">AAAJLHicjVZNj9s2EFW2X173K0mPvbC1FygCrWtvgiSXBbLYHnJoAXeRTQxYhkFRtE2YIgmSytol9Fd6bU/9Nb0URa/9HR3K8lqSbaAyDI/fvHnkDIekYsWZsf3+Xw9OPvjwo48/aZ22P/3s8y++fPjo8VsjM03oLZFc6lGMDeVM0FvLLKcjpSlOY07fxctr73/3nmrDpHhj14pOUjwXbMYItgBNHz4eTQfhaHoRRotEWoNGUwA7/V6/eNC+MSiNTlA+w+mj1kmUSJKlVFjCsTHjQV/ZicPaMsJp3o4yQxUmSzynYzAFTqmZuGLyOToDJEEzqeErLCrQaoTDqTHrNAZmiu3CNH0ePOazi/Te1Y4EvSMyTbFIXGQX9DXmc6mZ57gIaxwz4nZQ3q6LbR2M5Icd9YBxZmcvJ44JlVkqyCbNWcaRlcivA0qYpsTyNRiYgAAjiCxgFsTCatWUkvdMmbJmq03RYChIQGoKKa45dYpjJvIiwxJ35W/u3pTG2MB4sOaTGk1pqaRh3pGPS3DihhW0RucU6lch/lj8r1GI1JJzrNcV2vU91ph4QmdMHBingu9UfjhCho7LUtUgX+3AxphgYL2sS5TYLvymJNW7homEKgTtoqhW0TfFx5t5nUZYPn46cZ0BKiM6FyjidGYj2IoWdZ5Gms0XttcIE5U4IW2pXRnmuE6tJS1b/lJvUpPFBG+LUcHTjFum5V1+cFO1m6jfoKahAEQrJTcN9sqARRsJKkVy57rR0LDu+fA6jw6ycGZlilXursD46Wp4z6rTTJE71n767klRowrS4PqildTo3vYxT5pMcPnvVjKN5cqdn5/nJbnBJkz7E25D7u7YXbnXEp64pcsd8zBxMwVZHf7QTKvDH1W0mPGN3o6zlwmpVlM2qnmwpuRoTfdS8t6afq1S6Gj7eouzWMPB4YpQE0K/abYKFbZwTgqzzzMLrKgJYxCba5mJBDhnR0gJhcOquAhNWOoTzElp91JqcThjNtyehUzMvZhXAj3XPjtVkgn7fa84WNAlcpsFCVGi8V2ImBBUI8jsEq7PZyQFMc5DIRMK15yAe8GaSwc3zZk/oraip17j/0vuq+3XpDb/aqnnGqsFI6tmNyxhc7ru0m9S1FjMm3w8mMAe9hsfiuVu8s6gC/Hw1jBoviPsG28veoPnvf7PzzqvnpfvD63g6+Db4LtgELwIXgWvg2FwG5BgFfwa/Bb83vqj9Wfr79Y/G+rJgzLmq6D2tP79D3XHWiw=</latexit>
<latexit
X
p(zn |✓)p(xn |zn , ✓)

sha1_base64="ao3immhHoZ7Vujra3jqPoyMwfeE=">AAAJHnicjVZNj9s2EFU2Tet1v/JxzIWNvUARaF07CdJeFshic8ihBZxFdhPAMgKKom3WFEmQVNYOof+Qa3vKr8kt6LX9Nx3K8lqSbSAyDI/fvHnkDIekYsWZsf3+fzcObn516+tvWoftb7/7/ocfb9+5e2lkpgm9IJJL/SbGhnIm6IVlltM3SlOcxpy+judn3v/6HdWGSfHKLhUdp3gq2IQRbAG6jOyMWvz2dqff6xcP2jYGpdEJymf49k7rIEokyVIqLOHYmNGgr+zYYW0Z4TRvR5mhCpM5ntIRmAKn1IxdMd0cHQGSoInU8BUWFWg1wuHUmGUaAzPFdmaaPg/u89lZeu1qR4JeEZmmWCTOZ/oC86nUzHNchDWOGXEbKG/XxdYORvLdjnrAKLOT38aOCZVZKsgqzUnGkZXIVx4lTFNi+RIMTECAEURmMAtiYX1qSsk7pkxZs8WqaDAUJCA1hRSXnDrFMRN5kWGJu/I3d69KY2RgPFjlcY2mtFTSMO/IRyU4dsMKWqNzCvWrEH8v/tcoRGrJOdbLCu3sGmtMPKETJnaMU8E3Ks/3kKHjslQ1yKcbsDEmGFjP6xIltgk/L0n1rmEioQpBuyiqVfRT8fFmXqcRlo8ej11ngMqIziMUcTqxEWw+izqPI82mM9trhIlKnJC21K4Ms1+n1pKWzd/Xm9RkMcHrYlTwNOOWaXmV79xU7SbqN6hpKADRSslNg70wYNFGgkqR3LluNDSsezw8y6OdLJxZmWKVu1Mw/jgdXrPqNFPkjrWfvntY1KiCNLi+aCU1urZ9zMMmE1z+u5ZMY7lwx8fHeUlusAnT/oRbkbsbdldutYQnrulyw9xNXE1BVoffNdPq8HsVLWZ8pbfhbGVCqtWUjWrurCnZW9OtlLy3pl+rFNrbvt7iLNZwcLgi1ITQb5otQoUtnJPCbPPMDCtqwhjEplpmIgHO0R5SQuGwKq4+E5b6BHNS2r0ULsNwwmy4PguZmHoxrwR6rn10qCQT9pdecbCgE+RWCxKiROOrEDEhqEaQ2Qlcn09ICmKch0ImFK45AfeCNScObpojf0StRQ+9xpdLbqtt16Q2/2qppxqrGSOLZjfMYXO67txvUtRYzPN8NBjDHvYbH4rlzvPOoAvx8NYwaL4jbBuXj3qDp73+yyedZ0/L94dWcD94EPwcDIJfg2fBi2AYXAQk+DP4EPwV/N362PrU+tz6Z0U9uFHG3AtqT+vf/wGcdlYJ</latexit>
<latexit
that maximizes the data likelihood, i.e., the MLE estimate.

Using conditional independence of X1 , X2 , . . . Xn, we have

sha1_base64="iJ/41OzKI8gaaJ9TtDdMb6j1jvE=">AAAJHHicjVZNj9s2EFW2X173K2mPvbC1FygCrWMnQdrLAllsDzm0gLvIJgEsI6Ao2iZMkQRJZe0S+gu9tqf+mt6KXgv033Qo02tJtoHKMDx+8+aRMxySShVnxg6H/947ee/9Dz78qHPa/fiTTz/7/P6DL14ZWWhCb4jkUr9JsaGcCXpjmeX0jdIU5ymnr9Pllfe/fke1YVK8tGtFpzmeCzZjBFsPJcqwt/d7w8GwetC+MQpGLwrP+O2DzkmSSVLkVFjCsTGT0VDZqcPaMsJp2U0KQxUmSzynEzAFzqmZumqyJToDJEMzqeErLKrQeoTDuTHrPAVmju3CtH0ePOazi/zO1U0EvSUyz7HIXGIX9AXmc6mZ57gEa5wy4nZQ2W2KbR2MlIcdzYBJYWffTx0TqrBUkE2as4IjK5GvO8qYpsTyNRiYgAAjiCxgFsTC6jSUsndMmVCz1aZoMBQkIDWFFNecOsUxE2WVYcBd+C3dy2BMDIwHazxt0JSWShrmHeUkgFM3rqENOqdQvxrxx+p/g0Kklpxjva7Rru6w1sQzOmPiwDg1fKfywxEydFyRqxb5cge2xgQD62VTImC78OtAanYNExlVCNpFUa2Sr6uPN8smjbBy8mTqeiMUInqPUcLpzCaw9SzqPUk0my/soBUmanFC2qBdG+a4TqMlLVv+0mxSU6QEb4tRw/OCW6blbXlwU3XbqN+gpqUARCslNy32yoBFWwkqRUrn+snYsP75+KpMDrJwYWWOVekuwfjpcnzHatJMlTvWfvruYVWjGtLi+qIFanJn+5iHbSa4/Hcrmady5c7Pz8tAbrEJ0/6E25D7O3Zf7rWEJ27pcsc8TNxMQdaHPzTT+vBHFS1mfKO34+xlQurVlK1qHqwpOVrTvZS8t6HfqBQ62r7e4izVcHC4KtTE0G+arWKFLZyTwuzzzAIrauIUxOZaFiIDztkRUkbhsKouPhMHfYI5CfYgpxbHM2bj7VnIxNyLeSXQc92zUyWZsI8G1cGCLpDbLEiMMo1vY8SEoBpBZhdwfT4lOYhxHguZUbjmBNwL1lw4uGnO/BG1FT31Gv9fcl9tvyaN+ddLPddYLRhZtbthCZvT9Zd+k6LWYl6Xk9EU9rDf+FAsd132Rn2Ih7eGUfsdYd949XgwejYY/vy09/xZeH/oRF9F30TfRqPou+h59CIaRzcRiRbRr9Fv0e+dPzp/dv7q/L2hntwLMV9Gjafzz39hWFUr</latexit>
<latexit
ELBOn  log(p(D|✓))
Consider the latent variable model. We want to find parameter

sha1_base64="KTRdfC29A8dQetXdWvPDkBdGUTY=">AAAN33icjVdbj9tEFE5LF7qBhRYeeRnIrtQWJ8QplEpVUKtWqA+A0qo3NY5W4/EkGa09Y81MNt5YfuYN8cr/4U/wbzgznmxsJ1nqarVnz/nOd+Zc5tIwjZnS/f6/165/dOPg409uHrY//ezo8y9u3f7yjRILSehrImIh34VY0Zhx+lozHdN3qaQ4CWP6Njx7auxvz6lUTPBX+iKlkwTPOJsygjWoTm/f+CcI6YzxXLOzVcqIXkhajNsIPi4iiiJYAuaEDv3ejyTxnEEmOLb273tKX8R0mEcSLz3CJIlpUaJWewFeirWmkg+BSM8RxUojs37lPCOh1bYzoDktEDpBv9MlsgY0FbJEL5mei4VGZQDVnliiE/QLk0CexphQtyCFGEcYSbG0kMDoxqVpgu6s/LsIa3Sn7/Xvovx4deofF4+2gB6SbDbXw37vIUmQmKKVb3wHpcug7nKZzbbXwHgZu3EMiJE+LJxBGl9eRuRrL5f1yOar51RjRCiHWtMI4VCcW+W6EGUc6MW8zLvbLRlt5IRFS3zhlV6u+dXGQ3TLb1dupZ1ryFzNQxqLZT2wDVNh9ErQ0K+UNPMNf9bswhVepqTZoPTa34i6j6tmdlUnrgjJjS8vQzY78QwGF2EJ06bQVIrENUWLei1gjikmcxQwM535yvdWA2/FC2u1CLMDxt2fN2U33QrY3aujrUce4mXVeBu2deOh0o+2bANnG+ywcWfj/7OEy4SzKxLOfC8beNmHJdwOKI9q51X79Fan3+vbD20LvhM6LfeNTm/fvB5EgiwS2B0kxkqN/X6qJzmWmpkjrB0sFE0xOcMzOgaR44SqSW7P2wKdgCayh89UcDh0jLbqkeNEqYskBGQC20s1bUa5z6bnyaWpHXC6JCJJMCRsdtlzHM+EZAaTB1jikJF8oyradbK1gZFit6HuMF7o6cNJzni60JSTMs3pIjbNM1cH3AWSEh1fgIAJEDCCyBxWQeCAqeUxjs5ZqlzNsrJoEAoSgLYn9ujO4Uxm0G+TodPn7neRv3LCWEE8uKYmNVgqRSoUM4Zi7JSTfFTR1uAxhfpVgL/av2sQIqSIYywvKrCnl7rGwiM6ZXxHnIp+w/JsDxgmbpGkDfCTjbIREwQsz+oUTrdxf+lA9alhPKIpgnFJqUyDb+w/IxZ1GGHF+P4k7/jIeXQGKIjpVAfwetCocz+wd1Cv4cYrflxox10Js5+nNpJmN9eHVC1CgtfFqOiTRawZHC/Fzk3VbmrNBlUNBgBqIWLVQGcKJNpIME1JkcOVMFLsuDt6WgQ7UXihRYLTIn8Cwm9PRpeoOkzZ3O3pWOT3bI0qmgbWFM1Bg0vZ+NxrIsFkftaUSSiyvNvtFg7cQLtHWgk+3qCPxdZIGOAaLjbI3cByCaIaftdKq+H3MmrM4pJvg9nKhFSrKRrV3FlTsremWykZa42/Vim0d3yNFLNQwsGRl9egB/MmWbZ+9KptnJrjlCovBLKZFAseAeZkDyiicFjZt7vyHD/BMXFyL4Gr0psy7a3PQsZnhswwAV/ePjlMBePavazREOXuTY7sAx1uY04lgszgtdn/AR59UxbHnv2vAIGtBNelGuZw05yYI2pNemg4Ppxym227JrX1V0s9kzidM5I1p+EMNmd+fGY2KWo082Ux9iewh83Gh2LlL4sOPCjb5tXgN98I28KbQc9/0Ou/GHQeP3Dvh5utr1vftu60/NZPrcet561R63WLHHx38OLg/cH4CB/9cfTn0V8l9Po15/NVq/Yd/f0flzXK0w==</latexit>
<latexit
z1

x1

sha1_base64="fc4dQMV4STP0IGTGIqIJdYdmIGw=">AAAJKXicjVZNj9s2EFW2X173I5v02Atbe4E00Lp2EqS9LJDF9pBDC7iLbBLAMgyKom3CFMmQVNYOoV/Sa3Lqr+mt7bV/pENZXkuyDVSG4fGbN4+c4ZBUrDgztt//+87RRx9/8ulnreP25198+dXdk3v3XxqZaUKvieRSv46xoZwJem2Z5fS10hSnMaev4sWl9796S7VhUrywK0XHKZ4JNmUEW4AmJ3ffTFykDJuI/MG7ifh+ctLp9/rFg3aNQWl0gvIZTu61jqJEkiylwhKOjRkN+sqOHdaWEU7zdpQZqjBZ4BkdgSlwSs3YFTPP0SkgCZpKDV9hUYFWIxxOjVmlMTBTbOem6fPgIZ+dp7eudiToDZFpikXiIjunzzGfSc08x0VY45gRt4Xydl1s42Ak3++oB4wyO/1p7JhQmaWCrNOcZhxZifwioIRpSixfgYEJCDCCyBxmQSwsVU0pecuUKWu2XBcNhoIEpKaQ4opTpzhmIi8yLHFX/ubuRWmMDIwHCz6u0ZSWShrmHfmoBMduWEFrdE6hfhXiL8X/GoVILTnHelWhXd5ijYkndMrEnnEq+Fbl5wNk6LgsVQ3yxRZsjAkG1ou6RIltw69KUr1rmEioQtAuimoVfVt8vJnXaYTlo8dj1xmgMqLzCEWcTm0E+9CizuNIs9nc9hphohInpC21K8Mc1qm1pGWLd/UmNVlM8KYYFTzNuGVa3uR7N1W7ifoNahoKQLRSctNgLw1YtJGgUiR3rhsNDeueDS/zaC8LZ1amWOXuAoxfL4a3rDrNFLlj7afvHhY1qiANri9aSY1ubR/zsMkEl/9uJNNYLt3Z2VlekhtswrQ/4dbk7pbdlTst4Ykbutwy9xPXU5DV4ffNtDr8QUWLGV/rbTk7mZBqNWWjmntrSg7WdCcl763p1yqFDravtziLNRwcrgg1IfSbZstQYQvnpDC7PDPHipowBrGZlplIgHN6gJRQOKyKW9CEpT7BnJR2L6UWh1Nmw81ZyMTMi3kl0HPt02MlmbA/9IqDBZ0jt16QECUa34SICUE1gszO4fp8QlIQ4zwUMqFwzQm4F6w5d3DTnPojaiN67DX+v+Su2m5NavOvlnqmsZozsmx2wwI2p+su/CZFjcW8ykeDMexhv/GhWO4q7wy6EA9vDYPmO8Ku8fJRb/C01//tSefZ0/L9oRV8E3wXPAgGwY/Bs+B5MAyuAxJkwe/B++BD64/Wn62/Wv+sqUd3ypivg9rT+vc/JJlZ9w==</latexit>
<latexit
z2

x2

M step: Maximize ELBO wrt. model parameters ✓ for fixed variational parameters

sha1_base64="iJ/41OzKI8gaaJ9TtDdMb6j1jvE=">AAAJHHicjVZNj9s2EFW2X173K2mPvbC1FygCrWMnQdrLAllsDzm0gLvIJgEsI6Ao2iZMkQRJZe0S+gu9tqf+mt6KXgv033Qo02tJtoHKMDx+8+aRMxySShVnxg6H/947ee/9Dz78qHPa/fiTTz/7/P6DL14ZWWhCb4jkUr9JsaGcCXpjmeX0jdIU5ymnr9Pllfe/fke1YVK8tGtFpzmeCzZjBFsPJcqwt/d7w8GwetC+MQpGLwrP+O2DzkmSSVLkVFjCsTGT0VDZqcPaMsJp2U0KQxUmSzynEzAFzqmZumqyJToDJEMzqeErLKrQeoTDuTHrPAVmju3CtH0ePOazi/zO1U0EvSUyz7HIXGIX9AXmc6mZ57gEa5wy4nZQ2W2KbR2MlIcdzYBJYWffTx0TqrBUkE2as4IjK5GvO8qYpsTyNRiYgAAjiCxgFsTC6jSUsndMmVCz1aZoMBQkIDWFFNecOsUxE2WVYcBd+C3dy2BMDIwHazxt0JSWShrmHeUkgFM3rqENOqdQvxrxx+p/g0Kklpxjva7Rru6w1sQzOmPiwDg1fKfywxEydFyRqxb5cge2xgQD62VTImC78OtAanYNExlVCNpFUa2Sr6uPN8smjbBy8mTqeiMUInqPUcLpzCaw9SzqPUk0my/soBUmanFC2qBdG+a4TqMlLVv+0mxSU6QEb4tRw/OCW6blbXlwU3XbqN+gpqUARCslNy32yoBFWwkqRUrn+snYsP75+KpMDrJwYWWOVekuwfjpcnzHatJMlTvWfvruYVWjGtLi+qIFanJn+5iHbSa4/Hcrmady5c7Pz8tAbrEJ0/6E25D7O3Zf7rWEJ27pcsc8TNxMQdaHPzTT+vBHFS1mfKO34+xlQurVlK1qHqwpOVrTvZS8t6HfqBQ62r7e4izVcHC4KtTE0G+arWKFLZyTwuzzzAIrauIUxOZaFiIDztkRUkbhsKouPhMHfYI5CfYgpxbHM2bj7VnIxNyLeSXQc92zUyWZsI8G1cGCLpDbLEiMMo1vY8SEoBpBZhdwfT4lOYhxHguZUbjmBNwL1lw4uGnO/BG1FT31Gv9fcl9tvyaN+ddLPddYLRhZtbthCZvT9Zd+k6LWYl6Xk9EU9rDf+FAsd132Rn2Ih7eGUfsdYd949XgwejYY/vy09/xZeH/oRF9F30TfRqPou+h59CIaRzcRiRbRr9Fv0e+dPzp/dv7q/L2hntwLMV9Gjafzz39hWFUr</latexit>
<latexit
···
···
zn

xn

E step: Maximize ELBO wrt. variational parameters to find best q n (zn ) for fixed model par.

sha1_base64="ao3immhHoZ7Vujra3jqPoyMwfeE=">AAAJHnicjVZNj9s2EFU2Tet1v/JxzIWNvUARaF07CdJeFshic8ihBZxFdhPAMgKKom3WFEmQVNYOof+Qa3vKr8kt6LX9Nx3K8lqSbSAyDI/fvHnkDIekYsWZsf3+fzcObn516+tvWoftb7/7/ocfb9+5e2lkpgm9IJJL/SbGhnIm6IVlltM3SlOcxpy+judn3v/6HdWGSfHKLhUdp3gq2IQRbAG6jOyMWvz2dqff6xcP2jYGpdEJymf49k7rIEokyVIqLOHYmNGgr+zYYW0Z4TRvR5mhCpM5ntIRmAKn1IxdMd0cHQGSoInU8BUWFWg1wuHUmGUaAzPFdmaaPg/u89lZeu1qR4JeEZmmWCTOZ/oC86nUzHNchDWOGXEbKG/XxdYORvLdjnrAKLOT38aOCZVZKsgqzUnGkZXIVx4lTFNi+RIMTECAEURmMAtiYX1qSsk7pkxZs8WqaDAUJCA1hRSXnDrFMRN5kWGJu/I3d69KY2RgPFjlcY2mtFTSMO/IRyU4dsMKWqNzCvWrEH8v/tcoRGrJOdbLCu3sGmtMPKETJnaMU8E3Ks/3kKHjslQ1yKcbsDEmGFjP6xIltgk/L0n1rmEioQpBuyiqVfRT8fFmXqcRlo8ej11ngMqIziMUcTqxEWw+izqPI82mM9trhIlKnJC21K4Ms1+n1pKWzd/Xm9RkMcHrYlTwNOOWaXmV79xU7SbqN6hpKADRSslNg70wYNFGgkqR3LluNDSsezw8y6OdLJxZmWKVu1Mw/jgdXrPqNFPkjrWfvntY1KiCNLi+aCU1urZ9zMMmE1z+u5ZMY7lwx8fHeUlusAnT/oRbkbsbdldutYQnrulyw9xNXE1BVoffNdPq8HsVLWZ8pbfhbGVCqtWUjWrurCnZW9OtlLy3pl+rFNrbvt7iLNZwcLgi1ITQb5otQoUtnJPCbPPMDCtqwhjEplpmIgHO0R5SQuGwKq4+E5b6BHNS2r0ULsNwwmy4PguZmHoxrwR6rn10qCQT9pdecbCgE+RWCxKiROOrEDEhqEaQ2Qlcn09ICmKch0ImFK45AfeCNScObpojf0StRQ+9xpdLbqtt16Q2/2qppxqrGSOLZjfMYXO67txvUtRYzPN8NBjDHvYbH4rlzvPOoAvx8NYwaL4jbBuXj3qDp73+yyedZ0/L94dWcD94EPwcDIJfg2fBi2AYXAQk+DP4EPwV/N362PrU+tz6Z0U9uFHG3AtqT+vf/wGcdlYJ</latexit>
<latexit

Variational Inference

q* =argmin DKL(q(z)||p(z|x))

qt

qinit

In general, the “best” distribution we can model for Z using a given


paramterization may not exactly recover true posterior.
sha1_base64="WNCbvjd52pNu3oXMVZUJsTOVhJY=">AAAK53iclVZfb9s2EJe7f4n2r9ke98LNzuAUjme3xTagC9A0K9CHDPOCpi1geQZF0TZhitRIqnFC6DPsbdjrPsc+yR73un2JHWU5lmR72BQbOd/97ne80/HIMOFMm17vj8adN9586+139vb9d997/4MP7x589ELLVBF6SSSX6lWINeVM0EvDDKevEkVxHHL6MpyfOfvL11RpJsVzc53QUYyngk0YwQZU44PG80DgkOOxDcyMGpyhwNCFsU/Pn3yfndRt5+2l1AmSGTtqL44+P0Eb/jE2szC0T7Ox/WnsgO2bIGYRWhyBMWRTPgy4nKJkvPRoLzo3Ryh4dBw8Qrmh5uRc1CgIfBfrX7mf5NzV5QSf5iHb/z/kkeNbBg5wkii5qKcadDZYf7Q6O+r647vNXreXP2hT6BdC0yuewfhg7yaIJEljKgzhWOthv5eYkcXKMMJp5geppgkmczylQxAFjqke2fz9Z+gQNBGaSAVfYVCuLXtYHGt9HYeAdAXUdZtT7rKZWXxr8gNBr4iMYyyivAjPMJ9KxRzGBljhkBG7VmV+lWxlYCTbbqg6DFMz+XpkmUhSQwVZpjlJOTISuVZGEVOUGH4NAiZAwAgiM1gFMdDwFaboNUt0UbPFsmgQChKQikKK15zahGMmsjzDQm+L/5l9XghDDfFg24wqMGiNRGrmDNmwUI7soKStwDmF+pWA5/nvCoRIJTnH6roEO7vV1RYe0QkTW+KU9GuWb3eAoePSOKmBT9fKWkwQsJpXKQrd2v2iAFW7homIJgjaJaEqgd3p/pyYVWGEZcMHI9vso8KjeR+2Kp2YAKaZQc0HgWLTmenW3ETJT0hTcJfC7OaptKRh85tqk+o0JHhVjJI+TrlhSl5lWzeVX9e6DaprDAA0UnJdQy80SLSWYJKQzNpWMNCsdTw4y4KtKJwaGeMks6cgfHc6uEVVYTrPHSu3fHsvr1FJU8O6ohXQ4FZ2PvfqSDC574oyDuXCHh8fZwW4hiZMuQm3BLfW6JbcaAkHXMHlGrkduFyCLIffttJy+J2MBjO+5FtjNjIh5WrKWjW31pTsrOlGSs5a4a9UCu1sXydxFioYHK7RFFt0EmxgQAq9CdAznFDdCYFlqmQqoi0YgjmM78MdrhGF2ZVfLXQnX6nuOIdC7sbu7jBhprMajUxMHZljAj7rH+4nkgnzRTefM+gE2eX76aBI4asOYkJQhSDREzhNH5IYyDjvCBlROPUEHBNGn1g4eA7dxFqR7juO/065ybZZhcr6y5WfKgw3CbKoN8cc9qptzd2eRbV3e5EN+yPY0m4OQLHsRdbst+r+y4tPnEOXNzQX1U0Te5lZZ7HwEeBaP3ZDKecGhxrOz/yKIBiBfEEDxVAp7BN3W+nX7yabwov73f6X3d4PD5uPvynuLXveJ95nXtvre195j71n3sC79Ejj98afjb8af/vM/9n/xf91Cb3TKHw+9iqP/9s/GM39tA==</latexit>
<latexit
sha1_base64="7EMrRXBdYg2oEX7X5nDZLs4kwEo=">AAAJlHicjVbNjts2EFbSv7X7k6QFeumFrb1AEWhdOwmSHrJAFtsCObSFu8gmASwjoCjaJkyRBEll7RB6hV7bV+vbdCjLa4m2gcoWPJ755iNnOBwyVZwZOxz+e+fuRx9/8ulnJ53u5198+dW9+w++fm1koQm9JpJL/TbFhnIm6LVlltO3SlOcp5y+SZeX3v7mPdWGSfHKrhWd5ngu2IwRbL0qUQv27n5vOBhWD9oXRrXQi+pn/O7ByYckk6TIqbCEY2Mmo6GyU4e1ZYTTspsUhipMlnhOJyAKnFMzddVkS3QKmgzNpIZXWFRpmx4O58as8xSQObYLE9q88pjNLvJbUzcR9IbIPMcic4ld0JeYz6VmHuMSrHHKiNupym6bbGtgpDxsaDtMCjv7eeqYUIWlgmzCnBUcWYl83lHGNCWWr0HABAgYQWQBsyAWVqfFlL1nytQ5W22SBkNBAFJTCHHNqVMcM1FWEdZ6V/+W7lUtTAyMB2s8bcGUlkoa5g3lpFZO3bihbcE5hfw1gL9V/1sQIrXkHOt1A3Z5qwsmntEZEwfGaeh3LL8cAUPFFbkKwBc7ZTAmCFgv2xS1bud+VYPaVcNERhWCclFUq+T76uPFsg0jrJw8nrreCNUevUco4XRmE9h6FvUeJ5rNF3YQuImGn5C25m4Mc5ynVZKWLT+0i9QUKcHbZDT0ecEt0/KmPLipuqHWb1ATMADQSslNgF4ZkGgQoFKkdK6fjA3rn40vy+QgChdW5liV7gKE3y/Gt6g2zFSxY+2n7x5WOWpoAqxPWg1NbmXv8zBEgsm/W8o8lSt3dnZW1uAATZj2HW4D7u/QfblXEh64hcsd8jBwMwXZHP7QTJvDH2W0mPEN3w6zFwlpZlMG2TyYU3I0p3sheWuLv5UpdLR8vcRZqqFx+ELTbBUrbKFBCrMPMAusqIlTYJlrWYjsAIZgDu379IhrRqF3VeegiauZmtg71PIgpxbHM2bjbWtkYu7JPBPwue5pR0km7E+Dqs+gc+Q26xOjTOObGDEhqEYQ6Dmcpk9IDmScx0JmFE49AceENecODp5T37G2pB3P8f8p99n2s9CafzPzc43h/CersDiWsFddf+n3LArW9qqcjKawpX0fgGS5q7I36of+3pimeQVNLF1Z50f13cRdl85bHHwFuIbHbirl0uLUwPlZXREEIxAvaCAZuoB90oXbyii8m+wLrx8NRk8Hwz+f9F48r+8tJ9F30Q/Rj9Eoeha9iF5G4+g6ItEi+iv6O/qn823neeey8+sGevdO7fNN1Ho6f/wH53eB+Q==</latexit>
<latexit
Generative parameters
Gradient-based VI

model
descent/ascend to optimize ELBO?


h 

sha1_base64="3mMJ9tMsHpKlkc21Wye7UeP7mUU=">AAAJm3icjVZtjxo3EN6kbwd9S9qPVSW3cFIacRSSqO2HnpTT9UNUpRI9hSQSi05erwELr+3a3hycuz+jX9vf1X/T8bIcuwakLqwYZp557BmPx04UZ8YOBv/eu//e+x98+NFJq/3xJ59+9vmDh1+8NjLXhI6J5FK/TbChnAk6tsxy+lZpirOE0zfJ8tLb37yj2jApXtm1otMMzwWbMYItqCZ/XMdqwR7d/rn67vpBZ9AflA/aF4aV0ImqZ3T98OQ2TiXJMyos4diYyXCg7NRhbRnhtGjHuaEKkyWe0wmIAmfUTF055wKdgiZFM6nhFRaV2rqHw5kx6ywBZIbtwoQ2rzxms4vsztSOBb0hMsuwSF1sF/QF5nOpmce4GGucMOJ2qqLdJNsaGCkOG5oOk9zOfpo6JlRuqSCbMGc5R1Yin36UMk2J5WsQMAECRhBZwCyIhUVqMKXvmDJVzlabpMFQEIDUFEJcc+oUx0wUZYSV3lW/hXtVCRMD48FSTxswpaWShnlDMamUUzeqaRtwTiF/NeDL8n8DQqSWnGO9rsEu73TBxFM6Y+LAODX9juWXI2CouDxTAfhipwzGBAHrZZOi0u3crypQs2qYSKlCUC6KahV/U368WDRhhBWTp1PXGaLKo/MExZzObAw70KLO01iz+cL2AzdR8xPSVty1YY7zNErSsuVts0hNnhC8TUZNn+XcMi1vioObqh1q/QY1AQMArZTcBOiVAYkGASpFCue68ciw7tnosogPonBuZYZV4S5A+O1idIdqwkwZO9Z++u5xmaOaJsD6pFXQ+E72Po9DJJj8u6XMErlyZ2dnRQUO0IRp3+E24O4O3ZV7JeGBW7jcIQ8DN1OQ9eEPzbQ+/FFGixnf8O0we5GQejZlkM2DOSVHc7oXkrc2+BuZQkfL10ucJRoahy80zVY9hS00SGH2AWaBFTW9BFjmWuYiPYAhmEP7Pj3imlLoXeVxaHrlTE3PO1RyP6MW92bM9ratkYm5J/NMwOfapy0lmbDf98s+g86R26xPD6Ua3/QQE4JqBIGew2n6jGRAxnlPyJTCqSfgmLDm3MHBc+o71pa05Tn+P+U+234WGvOvZ36uMVwCyCosjiXsVddd+j2LgrW9KibDKWxp3wcgWe6q6Ay7ob83JklWQmNLV9b5UX03cePCeYuDrwDX8NhNpFxanBg4P8srgmAE4gUNJEPnsE/acFsZhneTfeH1k/7wh/7g92ed5z9X95aT6Kvo2+hRNIx+jJ5HL6JRNI5IJKO/or+jf1pfty5bv7ZebqD371U+X0aNpzX+D66QhM4=</latexit>
<latexit
⇡ r✓ log p✓ (x, z s ).
Gradient descent/ascend on variational parameters

Inference parameters φ model the posterior qφ (z|x)

r✓ ELBO = r✓ L(✓, )(x) = r✓ Eqφ (z|x) log p✓ (x, z)  log q (z | x)


= Eqφ (z|x) r✓ log p✓ (x, z)  log q (z | x)

i
Recall that ELBO is a lower bound to data likelihood. Can we use gradient
sha1_base64="WNCbvjd52pNu3oXMVZUJsTOVhJY=">AAAK53iclVZfb9s2EJe7f4n2r9ke98LNzuAUjme3xTagC9A0K9CHDPOCpi1geQZF0TZhitRIqnFC6DPsbdjrPsc+yR73un2JHWU5lmR72BQbOd/97ne80/HIMOFMm17vj8adN9586+139vb9d997/4MP7x589ELLVBF6SSSX6lWINeVM0EvDDKevEkVxHHL6MpyfOfvL11RpJsVzc53QUYyngk0YwQZU44PG80DgkOOxDcyMGpyhwNCFsU/Pn3yfndRt5+2l1AmSGTtqL44+P0Eb/jE2szC0T7Ox/WnsgO2bIGYRWhyBMWRTPgy4nKJkvPRoLzo3Ryh4dBw8Qrmh5uRc1CgIfBfrX7mf5NzV5QSf5iHb/z/kkeNbBg5wkii5qKcadDZYf7Q6O+r647vNXreXP2hT6BdC0yuewfhg7yaIJEljKgzhWOthv5eYkcXKMMJp5geppgkmczylQxAFjqke2fz9Z+gQNBGaSAVfYVCuLXtYHGt9HYeAdAXUdZtT7rKZWXxr8gNBr4iMYyyivAjPMJ9KxRzGBljhkBG7VmV+lWxlYCTbbqg6DFMz+XpkmUhSQwVZpjlJOTISuVZGEVOUGH4NAiZAwAgiM1gFMdDwFaboNUt0UbPFsmgQChKQikKK15zahGMmsjzDQm+L/5l9XghDDfFg24wqMGiNRGrmDNmwUI7soKStwDmF+pWA5/nvCoRIJTnH6roEO7vV1RYe0QkTW+KU9GuWb3eAoePSOKmBT9fKWkwQsJpXKQrd2v2iAFW7homIJgjaJaEqgd3p/pyYVWGEZcMHI9vso8KjeR+2Kp2YAKaZQc0HgWLTmenW3ETJT0hTcJfC7OaptKRh85tqk+o0JHhVjJI+TrlhSl5lWzeVX9e6DaprDAA0UnJdQy80SLSWYJKQzNpWMNCsdTw4y4KtKJwaGeMks6cgfHc6uEVVYTrPHSu3fHsvr1FJU8O6ohXQ4FZ2PvfqSDC574oyDuXCHh8fZwW4hiZMuQm3BLfW6JbcaAkHXMHlGrkduFyCLIffttJy+J2MBjO+5FtjNjIh5WrKWjW31pTsrOlGSs5a4a9UCu1sXydxFioYHK7RFFt0EmxgQAq9CdAznFDdCYFlqmQqoi0YgjmM78MdrhGF2ZVfLXQnX6nuOIdC7sbu7jBhprMajUxMHZljAj7rH+4nkgnzRTefM+gE2eX76aBI4asOYkJQhSDREzhNH5IYyDjvCBlROPUEHBNGn1g4eA7dxFqR7juO/065ybZZhcr6y5WfKgw3CbKoN8cc9qptzd2eRbV3e5EN+yPY0m4OQLHsRdbst+r+y4tPnEOXNzQX1U0Te5lZZ7HwEeBaP3ZDKecGhxrOz/yKIBiBfEEDxVAp7BN3W+nX7yabwov73f6X3d4PD5uPvynuLXveJ95nXtvre195j71n3sC79Ejj98afjb8af/vM/9n/xf91Cb3TKHw+9iqP/9s/GM39tA==</latexit>
<latexit
sha1_base64="ZeKYHnxwCdvtx/JqME7SP9AEx84=">AAAJpnicjVZfjxo3EN+kaXvQf5e2b3lxCyddI45CErV9yEk5XR/y0Fb0FHKRWHL1eg1YeG3X9uYAZ79LX9tv1G/T8bIcuwtIXVgxzPzmZ894PHakODO21/v33v0PHnz40cdHjeYnn372+RfHD798bWSqCR0SyaV+E2FDORN0aJnl9I3SFCcRp9fR/NLbr99RbZgUr+xS0XGCp4JNGMEWVDfHX6/emtCwBKE/b0I1Y6er94vvbo5bvW4vf9Cu0C+EVlA8g5uHR6swliRNqLCEY2NG/Z6yY4e1ZYTTrBmmhipM5nhKRyAKnFAzdvn0M3QCmhhNpIZXWJRryx4OJ8YskwiQCbYzU7d55SGbnSV3pmYo6C2RSYJF7EI7oy8xn0rNPMaFWOOIEbdVZc0q2cbASLbfUHUYpXby09gxoVJLBVmHOUk5shL5lUAx05RYvgQBEyBgBJEZzIJYWK8KU/yOKVPkbLFOGgwFAUhNIcQlp05xzESWR1joXfGbuVeFMDIwHqz6uAJTWippmDdko0I5doOStgLnFPJXAv6S/69AiNSSc6yXJdjlna428ZhOmNgzTkm/Zfn5ABgqLk1UDXyxVdbGBAHreZWi0G3drwpQtWqYiKlCUC6KahV+k3+8mFVhhGWjp2PX6qPCo/UEhZxObAib0aLW01Cz6cx2a26i5CekLbhLwxzmqZSkZfNVtUhNGhG8SUZJn6TcMi1vs72bqlnX+g1qagwAtFJyU0MvDEi0FqBSJHOuHQ4Ma58NLrNwLwqnViZYZe4ChF8vBneoKszksWPtp+8e5zkqaWpYn7QCGt7J3udxHQkm/24ok0gu3NnZWVaAa2jCtO9wa3B7i27LnZLwwA1cbpH7gespyPLw+2ZaHv4go8WMr/m2mJ1ISDmbspbNvTklB3O6E5K3VvgrmUIHy9dLnEUaGocvNM0WHYUtNEhhdgFmhhU1nQhYplqmIt6DIZhD+z454BpT6F35yWg6+UxNxzsUcjehFncmzHY2rZGJqSfzTMDnmicNJZmw33fzPoPOkVuvTwfFGt92EBOCagSBnsNp+owkQMZ5R8iYwqkn4Jiw5tzBwXPiO9aGtOE5/j/lLttuFirzL2d+qjFcAsiiXhxz2KuuPfd7FtXW9iob9cewpX0fgGS5q6zVb9f9vTGKkhwaWrqwzo/qu4kbZs5bHHwFuNaP3UjKucWRgfMzvyIIRiBe0EAydAr7pAm3lX79brIrvH7S7f/Q7f3+rPXieXFvOQoeBd8Gp0E/+DF4EbwMBsEwIMEq+Cv4O/incdr4rTFsXK+h9+8VPl8Flafxx38ZmIiJ</latexit>
<latexit
s
Gradient-based VI


h 

⇡ r✓ log p✓ (x, z s ).
estimated from data, even using a single sample:

where z ⇠ qφ (z|x) . Unbiased estimate of gradient and can be used in SGD.


r✓ ELBO = r✓ L(✓, )(x) = r✓ Eqφ (z|x) log p✓ (x, z)  log q (z | x)
= Eqφ (z|x) r✓ log p✓ (x, z)  log q (z | x)

i
Gradient with respect to generative parameters is easy and can be
sha1_base64="WNCbvjd52pNu3oXMVZUJsTOVhJY=">AAAK53iclVZfb9s2EJe7f4n2r9ke98LNzuAUjme3xTagC9A0K9CHDPOCpi1geQZF0TZhitRIqnFC6DPsbdjrPsc+yR73un2JHWU5lmR72BQbOd/97ne80/HIMOFMm17vj8adN9586+139vb9d997/4MP7x589ELLVBF6SSSX6lWINeVM0EvDDKevEkVxHHL6MpyfOfvL11RpJsVzc53QUYyngk0YwQZU44PG80DgkOOxDcyMGpyhwNCFsU/Pn3yfndRt5+2l1AmSGTtqL44+P0Eb/jE2szC0T7Ox/WnsgO2bIGYRWhyBMWRTPgy4nKJkvPRoLzo3Ryh4dBw8Qrmh5uRc1CgIfBfrX7mf5NzV5QSf5iHb/z/kkeNbBg5wkii5qKcadDZYf7Q6O+r647vNXreXP2hT6BdC0yuewfhg7yaIJEljKgzhWOthv5eYkcXKMMJp5geppgkmczylQxAFjqke2fz9Z+gQNBGaSAVfYVCuLXtYHGt9HYeAdAXUdZtT7rKZWXxr8gNBr4iMYyyivAjPMJ9KxRzGBljhkBG7VmV+lWxlYCTbbqg6DFMz+XpkmUhSQwVZpjlJOTISuVZGEVOUGH4NAiZAwAgiM1gFMdDwFaboNUt0UbPFsmgQChKQikKK15zahGMmsjzDQm+L/5l9XghDDfFg24wqMGiNRGrmDNmwUI7soKStwDmF+pWA5/nvCoRIJTnH6roEO7vV1RYe0QkTW+KU9GuWb3eAoePSOKmBT9fKWkwQsJpXKQrd2v2iAFW7homIJgjaJaEqgd3p/pyYVWGEZcMHI9vso8KjeR+2Kp2YAKaZQc0HgWLTmenW3ETJT0hTcJfC7OaptKRh85tqk+o0JHhVjJI+TrlhSl5lWzeVX9e6DaprDAA0UnJdQy80SLSWYJKQzNpWMNCsdTw4y4KtKJwaGeMks6cgfHc6uEVVYTrPHSu3fHsvr1FJU8O6ohXQ4FZ2PvfqSDC574oyDuXCHh8fZwW4hiZMuQm3BLfW6JbcaAkHXMHlGrkduFyCLIffttJy+J2MBjO+5FtjNjIh5WrKWjW31pTsrOlGSs5a4a9UCu1sXydxFioYHK7RFFt0EmxgQAq9CdAznFDdCYFlqmQqoi0YgjmM78MdrhGF2ZVfLXQnX6nuOIdC7sbu7jBhprMajUxMHZljAj7rH+4nkgnzRTefM+gE2eX76aBI4asOYkJQhSDREzhNH5IYyDjvCBlROPUEHBNGn1g4eA7dxFqR7juO/065ybZZhcr6y5WfKgw3CbKoN8cc9qptzd2eRbV3e5EN+yPY0m4OQLHsRdbst+r+y4tPnEOXNzQX1U0Te5lZZ7HwEeBaP3ZDKecGhxrOz/yKIBiBfEEDxVAp7BN3W+nX7yabwov73f6X3d4PD5uPvynuLXveJ95nXtvre195j71n3sC79Ejj98afjb8af/vM/9n/xf91Cb3TKHw+9iqP/9s/GM39tA==</latexit>
<latexit
sha1_base64="yhxkey2lbyrUnCSE3qyrFE31EJw=">AAAKnXicnVZtb9s2EJa7t0R7a7aP+zBudgqncDy7LbYBXYBmWYF+yAYvaJoClmFQFG0TpkiFpBonhH7Hvm4/a/9mR1mOJdnGhikxfL577jne6XhkmHCmTa/3d+PBe+9/8OFHe/v+x598+tnnDw++eKNlqgi9JJJL9TbEmnIm6KVhhtO3iaI4Djm9Cudnzn71jirNpHhtbhM6ivFUsAkj2IBqfNDwA4FDjsc2SGYsQ+ftwMyowR3kfh+1F0fo0QmqYoIYm1kY2pfZ2F6Pna59B0oWocVRhvwgZFM+DLicomS8ZGsvOndHKHh+HDxHuaHu5nzUKAj8R4Gg1/8SIfg5D1BZ0zJq+39EPXJ0atT1/fHDZq/byx+0KfQLoekVz2B8sHcXRJKkMRWGcKz1sN9LzMhiZRjhNPODVNMEkzme0iGIAsdUj2z+1jJ0CJoITaSCjzAo15Y9LI61vo1DQLpy6LrNKXfZzCy+N8EbpjdExjEWkXWVeYX5VCrmMDbACoeM2LUq86tkKwMj2XZD1WGYmsmPI8tEkhoqyDLNScqRkcg1IIqYosTwWxAwAQJGEJnBKoiBNq0wRe9YoouaLZZFg1CQgFQUUrzl1CYcM5HlGRZ6W3xn9nUhDDXEg2YfVWCJkonUzBmyYaEc2UFJW4FzCvUrAc/z3xUIkUpyjtVtCXZ2r6stPKITJrbEKenXLL/sAEPHpXFSA5+ulbWYIGA1r1IUurX7RQGqdg0TEU0QtEtCVRJ8k/85MavCCMuGT0e22UeFR/MJ7Dw6MQHMIIOaTwPFpjPTrbmJkp+QpuAuhdnNU2lJw+Z31SbVaUjwqhglfZxyw5S8ybZuKr+udRtU1xgAaKTkuoZeaJBoLcEkIZm1rWCgWet4cJYFW1E4NTLGSWZPQfj1dHCPqsJ0njtWbvn2cV6jkqaGdUUroMG97Hwe15Fgcp8VZRzKhT0+Ps4KcA1NmHITbglurdEtudESDriCyzVyO3C5BFkOv22l5fA7GQ1mfMm3xmxkQsrVlLVqbq0p2VnTjZSctcJfqRTa2b5O4ixUMDhcoym26CTYwIAUehOgZzihuhMCy1TJVERbMARzGN+HO1wjCrMrvxDoTr5S3XEOhdyN3X1gwkxnNRqZmDoyxwR81j/cTyQT5rtuPmfQCbLL99NBkcI3HcSEoApBoidwmj4jMZBx3hEyonDqCTgmjD6xcPAcuom1It13HP+dcpNtswqV9ZcrP1UYLgZkUW+OOexV25q7PYtq7/YiG/ZHsKXdHIBi2Yus2W/V/ZfXmDiHBoYujHVR3TSxl5l1Fgv/Alzrx24o5dzgUMP5mV8RBCOQL2igGCqFfeJuK/363WRTePOk2/++2/v9WfPFT8W9Zc/7yvvWa3t97wfvhffKG3iXHmlcN/5o/Nn4y//af+mf+78toQ8ahc+XXuXxr/4BGPrcmg==</latexit>
<latexit
sha1_base64="ZeKYHnxwCdvtx/JqME7SP9AEx84=">AAAJpnicjVZfjxo3EN+kaXvQf5e2b3lxCyddI45CErV9yEk5XR/y0Fb0FHKRWHL1eg1YeG3X9uYAZ79LX9tv1G/T8bIcuwtIXVgxzPzmZ894PHakODO21/v33v0PHnz40cdHjeYnn372+RfHD798bWSqCR0SyaV+E2FDORN0aJnl9I3SFCcRp9fR/NLbr99RbZgUr+xS0XGCp4JNGMEWVDfHX6/emtCwBKE/b0I1Y6er94vvbo5bvW4vf9Cu0C+EVlA8g5uHR6swliRNqLCEY2NG/Z6yY4e1ZYTTrBmmhipM5nhKRyAKnFAzdvn0M3QCmhhNpIZXWJRryx4OJ8YskwiQCbYzU7d55SGbnSV3pmYo6C2RSYJF7EI7oy8xn0rNPMaFWOOIEbdVZc0q2cbASLbfUHUYpXby09gxoVJLBVmHOUk5shL5lUAx05RYvgQBEyBgBJEZzIJYWK8KU/yOKVPkbLFOGgwFAUhNIcQlp05xzESWR1joXfGbuVeFMDIwHqz6uAJTWippmDdko0I5doOStgLnFPJXAv6S/69AiNSSc6yXJdjlna428ZhOmNgzTkm/Zfn5ABgqLk1UDXyxVdbGBAHreZWi0G3drwpQtWqYiKlCUC6KahV+k3+8mFVhhGWjp2PX6qPCo/UEhZxObAib0aLW01Cz6cx2a26i5CekLbhLwxzmqZSkZfNVtUhNGhG8SUZJn6TcMi1vs72bqlnX+g1qagwAtFJyU0MvDEi0FqBSJHOuHQ4Ma58NLrNwLwqnViZYZe4ChF8vBneoKszksWPtp+8e5zkqaWpYn7QCGt7J3udxHQkm/24ok0gu3NnZWVaAa2jCtO9wa3B7i27LnZLwwA1cbpH7gespyPLw+2ZaHv4go8WMr/m2mJ1ISDmbspbNvTklB3O6E5K3VvgrmUIHy9dLnEUaGocvNM0WHYUtNEhhdgFmhhU1nQhYplqmIt6DIZhD+z454BpT6F35yWg6+UxNxzsUcjehFncmzHY2rZGJqSfzTMDnmicNJZmw33fzPoPOkVuvTwfFGt92EBOCagSBnsNp+owkQMZ5R8iYwqkn4Jiw5tzBwXPiO9aGtOE5/j/lLttuFirzL2d+qjFcAsiiXhxz2KuuPfd7FtXW9iob9cewpX0fgGS5q6zVb9f9vTGKkhwaWrqwzo/qu4kbZs5bHHwFuNaP3UjKucWRgfMzvyIIRiBe0EAydAr7pAm3lX79brIrvH7S7f/Q7f3+rPXieXFvOQoeBd8Gp0E/+DF4EbwMBsEwIMEq+Cv4O/incdr4rTFsXK+h9+8VPl8Flafxx38ZmIiJ</latexit>
<latexit
s
Gradient-based VI


h 

h 

⇡ r✓ log p✓ (x, z s ).
estimated from data, even using a single sample:

r L(✓, )(x) = r Eqφ (z|x) log p✓ (x, z)  log q (z | x)


where z ⇠ qφ (z|x) . Unbiased estimate of gradient and can be used in SGD.
r✓ ELBO = r✓ L(✓, )(x) = r✓ Eqφ (z|x) log p✓ (x, z)  log q (z | x)


= Eqφ (z|x) r✓ log p✓ (x, z)  log q (z | x)

i

i
6= Eqφ (z|x) r log p✓ (x, z)  log q (z | x) .
Gradient with respect to generative parameters is easy and can be

Gradient with respect to inference parameters is not easy to estimate:


Reparameterization Trick
To take gradient with respect to inference parameters, for
continuous latent variables we can use reparameterization trick.
sha1_base64="yhxkey2lbyrUnCSE3qyrFE31EJw=">AAAKnXicnVZtb9s2EJa7t0R7a7aP+zBudgqncDy7LbYBXYBmWYF+yAYvaJoClmFQFG0TpkiFpBonhH7Hvm4/a/9mR1mOJdnGhikxfL577jne6XhkmHCmTa/3d+PBe+9/8OFHe/v+x598+tnnDw++eKNlqgi9JJJL9TbEmnIm6KVhhtO3iaI4Djm9Cudnzn71jirNpHhtbhM6ivFUsAkj2IBqfNDwA4FDjsc2SGYsQ+ftwMyowR3kfh+1F0fo0QmqYoIYm1kY2pfZ2F6Pna59B0oWocVRhvwgZFM+DLicomS8ZGsvOndHKHh+HDxHuaHu5nzUKAj8R4Gg1/8SIfg5D1BZ0zJq+39EPXJ0atT1/fHDZq/byx+0KfQLoekVz2B8sHcXRJKkMRWGcKz1sN9LzMhiZRjhNPODVNMEkzme0iGIAsdUj2z+1jJ0CJoITaSCjzAo15Y9LI61vo1DQLpy6LrNKXfZzCy+N8EbpjdExjEWkXWVeYX5VCrmMDbACoeM2LUq86tkKwMj2XZD1WGYmsmPI8tEkhoqyDLNScqRkcg1IIqYosTwWxAwAQJGEJnBKoiBNq0wRe9YoouaLZZFg1CQgFQUUrzl1CYcM5HlGRZ6W3xn9nUhDDXEg2YfVWCJkonUzBmyYaEc2UFJW4FzCvUrAc/z3xUIkUpyjtVtCXZ2r6stPKITJrbEKenXLL/sAEPHpXFSA5+ulbWYIGA1r1IUurX7RQGqdg0TEU0QtEtCVRJ8k/85MavCCMuGT0e22UeFR/MJ7Dw6MQHMIIOaTwPFpjPTrbmJkp+QpuAuhdnNU2lJw+Z31SbVaUjwqhglfZxyw5S8ybZuKr+udRtU1xgAaKTkuoZeaJBoLcEkIZm1rWCgWet4cJYFW1E4NTLGSWZPQfj1dHCPqsJ0njtWbvn2cV6jkqaGdUUroMG97Hwe15Fgcp8VZRzKhT0+Ps4KcA1NmHITbglurdEtudESDriCyzVyO3C5BFkOv22l5fA7GQ1mfMm3xmxkQsrVlLVqbq0p2VnTjZSctcJfqRTa2b5O4ixUMDhcoym26CTYwIAUehOgZzihuhMCy1TJVERbMARzGN+HO1wjCrMrvxDoTr5S3XEOhdyN3X1gwkxnNRqZmDoyxwR81j/cTyQT5rtuPmfQCbLL99NBkcI3HcSEoApBoidwmj4jMZBx3hEyonDqCTgmjD6xcPAcuom1It13HP+dcpNtswqV9ZcrP1UYLgZkUW+OOexV25q7PYtq7/YiG/ZHsKXdHIBi2Yus2W/V/ZfXmDiHBoYujHVR3TSxl5l1Fgv/Alzrx24o5dzgUMP5mV8RBCOQL2igGCqFfeJuK/363WRTePOk2/++2/v9WfPFT8W9Zc/7yvvWa3t97wfvhffKG3iXHmlcN/5o/Nn4y//af+mf+78toQ8ahc+XXuXxr/4BGPrcmg==</latexit>
<latexit

h 
Reparameterization Trick

r L(✓, )(x) = r Eqφ (z|x) log p✓ (x, z)  log q (z | x)



Recall we could not push gradient into the expectation.

i
6= Eqφ (z|x) r log p✓ (x, z)  log q (z | x) .
To take gradient with respect to inference parameters, for
continuous latent variables we can use reparameterization trick.
sha1_base64="yhxkey2lbyrUnCSE3qyrFE31EJw=">AAAKnXicnVZtb9s2EJa7t0R7a7aP+zBudgqncDy7LbYBXYBmWYF+yAYvaJoClmFQFG0TpkiFpBonhH7Hvm4/a/9mR1mOJdnGhikxfL577jne6XhkmHCmTa/3d+PBe+9/8OFHe/v+x598+tnnDw++eKNlqgi9JJJL9TbEmnIm6KVhhtO3iaI4Djm9Cudnzn71jirNpHhtbhM6ivFUsAkj2IBqfNDwA4FDjsc2SGYsQ+ftwMyowR3kfh+1F0fo0QmqYoIYm1kY2pfZ2F6Pna59B0oWocVRhvwgZFM+DLicomS8ZGsvOndHKHh+HDxHuaHu5nzUKAj8R4Gg1/8SIfg5D1BZ0zJq+39EPXJ0atT1/fHDZq/byx+0KfQLoekVz2B8sHcXRJKkMRWGcKz1sN9LzMhiZRjhNPODVNMEkzme0iGIAsdUj2z+1jJ0CJoITaSCjzAo15Y9LI61vo1DQLpy6LrNKXfZzCy+N8EbpjdExjEWkXWVeYX5VCrmMDbACoeM2LUq86tkKwMj2XZD1WGYmsmPI8tEkhoqyDLNScqRkcg1IIqYosTwWxAwAQJGEJnBKoiBNq0wRe9YoouaLZZFg1CQgFQUUrzl1CYcM5HlGRZ6W3xn9nUhDDXEg2YfVWCJkonUzBmyYaEc2UFJW4FzCvUrAc/z3xUIkUpyjtVtCXZ2r6stPKITJrbEKenXLL/sAEPHpXFSA5+ulbWYIGA1r1IUurX7RQGqdg0TEU0QtEtCVRJ8k/85MavCCMuGT0e22UeFR/MJ7Dw6MQHMIIOaTwPFpjPTrbmJkp+QpuAuhdnNU2lJw+Z31SbVaUjwqhglfZxyw5S8ybZuKr+udRtU1xgAaKTkuoZeaJBoLcEkIZm1rWCgWet4cJYFW1E4NTLGSWZPQfj1dHCPqsJ0njtWbvn2cV6jkqaGdUUroMG97Hwe15Fgcp8VZRzKhT0+Ps4KcA1NmHITbglurdEtudESDriCyzVyO3C5BFkOv22l5fA7GQ1mfMm3xmxkQsrVlLVqbq0p2VnTjZSctcJfqRTa2b5O4ixUMDhcoym26CTYwIAUehOgZzihuhMCy1TJVERbMARzGN+HO1wjCrMrvxDoTr5S3XEOhdyN3X1gwkxnNRqZmDoyxwR81j/cTyQT5rtuPmfQCbLL99NBkcI3HcSEoApBoidwmj4jMZBx3hEyonDqCTgmjD6xcPAcuom1It13HP+dcpNtswqV9ZcrP1UYLgZkUW+OOexV25q7PYtq7/YiG/ZHsKXdHIBi2Yus2W/V/ZfXmDiHBoYujHVR3TSxl5l1Fgv/Alzrx24o5dzgUMP5mV8RBCOQL2igGCqFfeJuK/363WRTePOk2/++2/v9WfPFT8W9Zc/7yvvWa3t97wfvhffKG3iXHmlcN/5o/Nn4y//af+mf+78toQ8ahc+XXuXxr/4BGPrcmg==</latexit>
<latexit

h 
Reparameterization Trick

respect to another noise that is independent from q.


r L(✓, )(x) = r Eqφ (z|x) log p✓ (x, z)  log q (z | x)

This is the basic idea behind reparameterization trick.



Recall we could not push gradient into the expectation.

i
6= Eqφ (z|x) r log p✓ (x, z)  log q (z | x) .
To take gradient with respect to inference parameters, for

What if we could replace expectation with an expectation with

Maybe we can write z as some function of independent noise.


continuous latent variables we can use reparameterization trick.
sha1_base64="Igwxxfz8s+sJMkpIB6lf68QKnaU=">AAAKZHicjVZfb9s2EJe7f4nWbemKPQ0YuNkB0sLx7LbY+rAADbIBfdgAL2jaApZnUDRtE6ZIjqQaJ5y+5N72BfY5dpTkWJJtrLINn+9+9zve+XhkrDgztt//p3Xvgw8/+viTg8Pw0/ufff7F0YMvXxuZakKviORSv42xoZwJemWZ5fSt0hQnMadv4uWFt795R7VhUryyN4qOEzwXbMYItqCaHP0dCRxzPHGRWrAMRQm2izh2v2QT92epPLn9a/UITJzO7AjNTgrMzN1mj1Ck2Xxhx+gM7edRJxFVhnEp3oPlf/zqUXayhJOjdr/Xzx+0LQxKoR2Uz3Dy4OA2mkqSJlRYwrExo0Ff2bHD2jLCaRZGqaEKkyWe0xGIAifUjF1e/Awdg2aKZlLDR1iUa6seDifG3CQxIP1iTdPmlftsdpHcmcJI0GsikwSLqYvsgr7EfC418xgXYY1jRtxGlYV1srWBkWy3oe4wSu3s+dgxoVJLBSnSnKUcWYl8H6Ep05RYfgMCJkDACCILWAWx0G01puk7pkxZs1VRNAgFCUhNIcUbTp3imIksz7DUu/I7c69KYWQgHvTsuAZTWippmDdko1I5dsOKtgbnFOpXAf6a/65BiNSSc6xvKrCLO11j4VM6Y2JHnIp+w/LzHjB0XJqoBvh8o2zEBAHrZZ2i1G3cL0tQvWuYmFKFoF0U1Sr6Nn95MavDCMtGT8euPUClR/tJsQEjGCUWtZ8WW63XcBMVPyFtyV0Js5+n1pKWLW/rTWrSmOB1MSr6JOWWaXmd7dxUYVPrN6hpMADQSslNA70yINFGgkqRzLlONDSsczq8yKKdKJxamWCVuXMQfjsf3qHqMJPnjrVfvnuc16iiaWB90UpodCd7n8dNJJj8Z02ZxHLlTk9PsxLcQBOm/YQrwJ0NuiO3WsID13C5Qe4GFkuQ1fC7VloNv5fRYsYLvg1mKxNSraZsVHNnTcnemm6l5K01/lql0N729RJnsYbB4RtNs1VXYQsDUphtgFlgRU03Bpa5lqmY7sAQzGF8H+9xnVKYXfm5brr5Sk3XO5RyL6EWd2fMdtejkYm5J/NMwOfC40MlmbDf9/I5A6exK/6fLppqfN1FTAiqESR6BqfpM5IAGeddIacUTj0Bx4Q1Zw4OnmM/sdakh57j/Sm32barUFt/tfJzjeFeQFbN5ljCXnWdpd+zqPHfXmajwRi2tJ8DUCx3mbUHnaZ/cSlJcmhk6co6H9VPE3eVOW9x8Bbg2jx2YymXFscGzs/8iiAYgXxBA8XQKewTf1sZNO8m28LrJ73BD73+78/aL34q7y0HwdfBd8FJMAh+DF4EL4NhcBWQ1vPWH615a3H4b3g/fBh+VUDvtUqfh0HtCb/5Dwczzws=</latexit>
<latexit sha1_base64="w11G8rruc3g1mlfri6ImUWo8zMU=">AAAKBXicjVbNbhs3EF4n/bHUnzjtsRe2koEkkFUpCdIeaiCGe8ihBVQjTgJoBYPLpSRCXJIlubFsguc+TW9Fr32OPE07XK2s3ZUEdC3Bo5lvPnJmhzNMFGfGDgYfDu7d/+jjTz49bLU/+/yLLx8cPfzqjZG5JvSSSC71uwQbypmgl5ZZTt8pTXGWcPo2WZwH+9v3VBsmxWt7o+gkwzPBpoxgC6qrIzO9crGdU4t7KFZz5h/FGbbzZOpu/WN0imIuZ0itMRvr0gO+gjxZIX8HZIMFYCxFG7/H7aujzqA/KB60LQxLoROVz+jq4eFtnEqSZ1RYwrEx4+FA2YnD2jLCqW/HuaEKkwWe0TGIAmfUTFyRHY+OQZOiqdTwFRYV2qqHw5kxN1kCyLBJ07QF5T6bnWd3pnYs6DWRWYZFWuTrFeYzqVnAuBhrnDDiNirfrpOtDYz43Ya6wzi30x8njgmVWyrIKsxpzpGVKLxolDJNieU3IGACBIwgModdEAvlUGNK3zNlypwtV0mDpSAAqSmEeMOpUxwz4YsIS70r/3v3uhTGBtaDoprUYEpLJQ0LBj8ulRM3qmhrcE4hfxXgL8XvGoRILTnH+qYCO7/TNTae0ikTO9ap6DcsP+8BQ8XlmWqAzzbKxpogYL2oU5S6jftFCapXDRMpVQjKRVGt4m+LvyD6OowwP342cZ0hKj06T+H80amN4axb1HkWazab237DTVT8hLQld2WZ/Ty1krRscVsvUpMnBK+TUdFnObdMy2u/81C1m9pwQE2DAYBWSm4a6KUBiTYCVIp457rxyLDuyejcxztROLcyw8q7MxB+PRvdoeowU8SOddi+e1LkqKJpYEPSSmh8JwefJ00kmMJ3TZklculOTk58CW6gCdOhw63A3Q26K7dKIgDXcLlB7gautiCry+/aaXX5vYwWM77i22C2IiHVbMpGNnfmlOzN6VZIwVrjr2UK7S3fIHGWaGgcodA0W/YUttAghdkGmDlW1PQSYJlpmYt0B4ZgDu37eI9rSqF3FYPX9Iqdml5wKOV+FqbwlNneujUyMQtkgQn4XPu4pSQT9vt+0WdgOrvV++mhVOPrHmJCUI0g0FOYps9JBmSc94RMKUw9AWPCmlMHg+c4dKw1aStw/H/KbbbtLNT2X838TGO4HZBlszgWcFZddxHOLGq82ws/Hk7gSIc+AMlyF74z7Db9i8tFkhXQ2NKldWHV0E3cpXfB4uAjwLU5dhMpFxYnBuZncUUQjEC8oIFk6BzOSbitDJt3k23hzdP+8EV/8NvzzsufynvLYfRN9F30KBpGP0Qvo1fRKLqMSPQh+vfg8KDV+qP1Z+uv1t8r6L2D0ufrqPa0/vkP8GatXA==</latexit>
<latexit
sha1_base64="uu2G3PNGw+dkj+s/+UzlbnSGsaQ=">AAAKqnicjVZfb9s2EJe7f4n2r9ke9rAXbXaAtlM8uy22AV2AFtlDB3SDFzRtB8sIKIq2CVMkR1KNE4L7NHvdvs++zY6yHEuyDUyJ4fPd737HOx2PTCWj2gwG/3buvPPue+9/cHAYfvjRx598evfos1daFAqTCyyYUG9SpAmjnFwYahh5IxVBecrI63Rx5u2v3xKlqeAvzbUkkxzNOJ1SjAyoLo86XyQ5MvN0am9clGiaR+VvjJj91SVfJSmdsXtJKlimr3P4skleuHgFUrnNKJq5pl3TWY7cfe+p7ofJk+SF4DNGpkbR2dwgpcRV8iSsRT2N2vzRN00VkZoywWGBIhMm2hEuDpM/CpSFe9zaea2yqpYwWOcD8s9ute5+eHm3O+gPyifaFoaV0A2qZ3R5dHCTZAIXOeEGM6T1eDiQZmKRMhQz4sKk0EQivEAzMgaRo5zoiS1foYuOQZNFU6Hgw01UauseFuXa5wVIv1TdtnnlPpuZ57emMOHkCos8RzyziZmT54jNhKIeYxOkUEqx3ahc2CRbGyh2uw1Nh3Fhpj9MLOWyMITjVZrTgkVGRL4bo4wqgg27BgFhIKA4wnNYBTbQsw2m7C2VuqrZclU0CAUJCEUgxWtGrGSIcldmWOlt9e3sy0oYa4gHnT9pwKQSUmjqDW5cKSd2VNM24IxA/WrAF+XvBgQLJRhD6roGO7vVtRaekSnlO+LU9BuWn/aAoeOKXLbAzzbKVkwQkFo0KSrdxv28AjW7hvKMyAjaRRIlYUb4Py+6JgxTN340sd1hVHl0H0aJHwQJDCQTdR8l5UTot9x4zY8LU3HXwuznabSkoYubZpPqIsVoXYyaPi+YoTCW3M5NFba1foPqFgMAjRBMt9BLDRJpJSgldtb2kpGmvZPRmUt2olBhRI6ks89A+OXZ6BbVhOnNVHX2QVmjmqaF9UWroMmt7H0etJFg8p81JczTpT05OXEVuIXGVPkJtwL3Nuie2GoJD1zDxQa5G7hagqiH37XSevi9jAZRtuLbYLYywfVqilY1d9YU763pVkrNMxD4G5WK9ravlxhNFQwO32iKLmOJDAxIrrcBeo4k0XEKLDMlCp7twMAhCOP7eI9rRmB2lbcDHZcr1bF3qOR+TgyKp9TE69FI+cyTeSbgs+HxoRSUm2/75ZyB092u3k8cZQpdxRHlnKgIEj2F0/QxzoGMsZiLjMCpx+GYMPrUwsFz7CfWmvTQc/x/ym227So01l+v/EwhOad42W6OBexV21v4PRu13u25Gw8nsKXXV4xz1x322v7lFSPNS2hiyNJYH9VPE3vhrLdY+Ofg2j52UyEWBqUazs/yisAphnxBA8VQBewTf1sZtu8m28Krh/3hd/3Bb4+7T3+s7i0HwZfB18G9YBh8HzwNngej4CLAnT87f3X+7vwTxuF5+Hs4XkHvdCqfz4PGE2b/AUoD7DM=</latexit>
<latexit

For example,

Then, we have
sha1_base64="mzSkkiMeDwxlWjvXo082//AnmOE=">AAAJ9XicjVbNjiM1EO5d/ibhbxaOXBqSkWZXmZDsroADI+1oQNoDSGG0s7tSOorcbiex4raN7d5JxvhRuCGuPAePwFNwhRvlTmfSP4lEJ61Uqr767CqXy44lo9oMBn/du//W2++8+95Rq/3+Bx9+9PHxg09eapEpTK6xYEK9jpEmjHJybahh5LVUBKUxI6/i5aW3v3pDlKaCvzBrSSYpmnM6oxgZUE2PcZQis4hj+72b2p+nkVzQ041qZm/dL1tx5R668axkeTgJz8OyrzyNiNSUCd5ATo87g/4gf8KmMCyETlA8o+mDo9soEThLCTeYIa3Hw4E0E4uUoZgR144yTSTCSzQnYxA5Some2DwbLjwBTRLOhIKXmzDXlj0sSrVepzEg/SR13eaVh2xmkd6Z2hEnN1ikKeKJjcyCPEdsLhT1GBshhWKK7U7l2lWyrYFit99QdRhnZvbNxFIuM0M43oQ5y1hoROgXNkyoItiwNQgIAwHFIV7ALLCB5a8wJW+o1EXOVpukwVAQgFAEQlwzYiVDlLs8wkJvi19nXxTCWMN4UESTCkwqIYWm3uDGhXJiRyVtBc4I5K8E/CH/X4FgoQRjSK1LsMs7XW3iCZlRvmeckn7H8t0BMFRclsoa+GKnrI0JAlLLKkWh27lfFaBq1VCeEBlCuUiiZPR5/vGiq8IwdeMnE9sZhoVH53EYMTIzEextE3aeRIrOF6Zfc+MlPy5MwV0a5jBPpSQNXd5Wi1RnMUbbZJT0acYMVeLG7d1U7brWb1BdYwCgEYLpGnqlQSK1AKXEztpuNNK0eza6dNFeFMqMSJF09gKEHy9Gd6gqTOexI+Wnbx/lOSppaliftAIa3cne51EdCSb/binTWKzs2dmZK8A1NKbKd7gNuLtDd0WjJDxwCxc75H7gZgqiPPy+mZaHP8hoEGUbvh2mEQkuZ1PUsrk3p/hgThsheWuFv5Kp8GD5eonRWEHj8IWm6KonkYEGyXUToBdIEt2LgWWuRMaTPRiMGLTvkwOuCYHelR+0upfPVPe8QyH3U2JQb0ZNb9saKZ97Ms8EfLZ90pKCcvNlP+8zcODazfr0wkShm15IOScqhEDP4TR9ilMgY6zHRULg1ONwTBh9buHgOfEda0va8hz/n7LJ1sxCZf7lzM8VggsFXtWLYwl71XaXfs+GtbW9cuPhBLa07wOQLHvlOsNu3X9z70hzaGTIylg/qu8m9tpZb7Hw5eBaP3ZjIZYGxRrOz/yKwCmGeEEDyVAZ7JM23FaG9btJU3j5uD/8qj/46Wnn2bfFveUo+Cz4IjgNhsHXwbPgeTAKrgMc/Bn8HfwT/Nu6af3a+q31+wZ6/17h82lQeVp//AcTkaqp</latexit>
<latexit

Reparameterization Trick

Eqφ (z|x) [f (z)] = Ep(✏) [f (z)]


z ⇠ N µ, diag() () z = µ + ✏ $ ,

Let f✓, (z) = log p✓ (x, z) − log q (z | x) , then



Write Z as a push-forward of some known noise term:

r Eqφ (z|x) [f (z)] = r Ep(✏) [f (z)] = Ep(✏) [r f (z)]



✏ ⇠ N 0, I .

[from law of unconscious statistician]


sha1_base64="Igwxxfz8s+sJMkpIB6lf68QKnaU=">AAAKZHicjVZfb9s2EJe7f4nWbemKPQ0YuNkB0sLx7LbY+rAADbIBfdgAL2jaApZnUDRtE6ZIjqQaJ5y+5N72BfY5dpTkWJJtrLINn+9+9zve+XhkrDgztt//p3Xvgw8/+viTg8Pw0/ufff7F0YMvXxuZakKviORSv42xoZwJemWZ5fSt0hQnMadv4uWFt795R7VhUryyN4qOEzwXbMYItqCaHP0dCRxzPHGRWrAMRQm2izh2v2QT92epPLn9a/UITJzO7AjNTgrMzN1mj1Ck2Xxhx+gM7edRJxFVhnEp3oPlf/zqUXayhJOjdr/Xzx+0LQxKoR2Uz3Dy4OA2mkqSJlRYwrExo0Ff2bHD2jLCaRZGqaEKkyWe0xGIAifUjF1e/Awdg2aKZlLDR1iUa6seDifG3CQxIP1iTdPmlftsdpHcmcJI0GsikwSLqYvsgr7EfC418xgXYY1jRtxGlYV1srWBkWy3oe4wSu3s+dgxoVJLBSnSnKUcWYl8H6Ep05RYfgMCJkDACCILWAWx0G01puk7pkxZs1VRNAgFCUhNIcUbTp3imIksz7DUu/I7c69KYWQgHvTsuAZTWippmDdko1I5dsOKtgbnFOpXAf6a/65BiNSSc6xvKrCLO11j4VM6Y2JHnIp+w/LzHjB0XJqoBvh8o2zEBAHrZZ2i1G3cL0tQvWuYmFKFoF0U1Sr6Nn95MavDCMtGT8euPUClR/tJsQEjGCUWtZ8WW63XcBMVPyFtyV0Js5+n1pKWLW/rTWrSmOB1MSr6JOWWaXmd7dxUYVPrN6hpMADQSslNA70yINFGgkqRzLlONDSsczq8yKKdKJxamWCVuXMQfjsf3qHqMJPnjrVfvnuc16iiaWB90UpodCd7n8dNJJj8Z02ZxHLlTk9PsxLcQBOm/YQrwJ0NuiO3WsID13C5Qe4GFkuQ1fC7VloNv5fRYsYLvg1mKxNSraZsVHNnTcnemm6l5K01/lql0N729RJnsYbB4RtNs1VXYQsDUphtgFlgRU03Bpa5lqmY7sAQzGF8H+9xnVKYXfm5brr5Sk3XO5RyL6EWd2fMdtejkYm5J/NMwOfC40MlmbDf9/I5A6exK/6fLppqfN1FTAiqESR6BqfpM5IAGeddIacUTj0Bx4Q1Zw4OnmM/sdakh57j/Sm32barUFt/tfJzjeFeQFbN5ljCXnWdpd+zqPHfXmajwRi2tJ8DUCx3mbUHnaZ/cSlJcmhk6co6H9VPE3eVOW9x8Bbg2jx2YymXFscGzs/8iiAYgXxBA8XQKewTf1sZNO8m28LrJ73BD73+78/aL34q7y0HwdfBd8FJMAh+DF4EL4NhcBWQ1vPWH615a3H4b3g/fBh+VUDvtUqfh0HtCb/5Dwczzws=</latexit>
<latexit sha1_base64="w11G8rruc3g1mlfri6ImUWo8zMU=">AAAKBXicjVbNbhs3EF4n/bHUnzjtsRe2koEkkFUpCdIeaiCGe8ihBVQjTgJoBYPLpSRCXJIlubFsguc+TW9Fr32OPE07XK2s3ZUEdC3Bo5lvPnJmhzNMFGfGDgYfDu7d/+jjTz49bLU/+/yLLx8cPfzqjZG5JvSSSC71uwQbypmgl5ZZTt8pTXGWcPo2WZwH+9v3VBsmxWt7o+gkwzPBpoxgC6qrIzO9crGdU4t7KFZz5h/FGbbzZOpu/WN0imIuZ0itMRvr0gO+gjxZIX8HZIMFYCxFG7/H7aujzqA/KB60LQxLoROVz+jq4eFtnEqSZ1RYwrEx4+FA2YnD2jLCqW/HuaEKkwWe0TGIAmfUTFyRHY+OQZOiqdTwFRYV2qqHw5kxN1kCyLBJ07QF5T6bnWd3pnYs6DWRWYZFWuTrFeYzqVnAuBhrnDDiNirfrpOtDYz43Ya6wzi30x8njgmVWyrIKsxpzpGVKLxolDJNieU3IGACBIwgModdEAvlUGNK3zNlypwtV0mDpSAAqSmEeMOpUxwz4YsIS70r/3v3uhTGBtaDoprUYEpLJQ0LBj8ulRM3qmhrcE4hfxXgL8XvGoRILTnH+qYCO7/TNTae0ikTO9ap6DcsP+8BQ8XlmWqAzzbKxpogYL2oU5S6jftFCapXDRMpVQjKRVGt4m+LvyD6OowwP342cZ0hKj06T+H80amN4axb1HkWazab237DTVT8hLQld2WZ/Ty1krRscVsvUpMnBK+TUdFnObdMy2u/81C1m9pwQE2DAYBWSm4a6KUBiTYCVIp457rxyLDuyejcxztROLcyw8q7MxB+PRvdoeowU8SOddi+e1LkqKJpYEPSSmh8JwefJ00kmMJ3TZklculOTk58CW6gCdOhw63A3Q26K7dKIgDXcLlB7gautiCry+/aaXX5vYwWM77i22C2IiHVbMpGNnfmlOzN6VZIwVrjr2UK7S3fIHGWaGgcodA0W/YUttAghdkGmDlW1PQSYJlpmYt0B4ZgDu37eI9rSqF3FYPX9Iqdml5wKOV+FqbwlNneujUyMQtkgQn4XPu4pSQT9vt+0WdgOrvV++mhVOPrHmJCUI0g0FOYps9JBmSc94RMKUw9AWPCmlMHg+c4dKw1aStw/H/KbbbtLNT2X838TGO4HZBlszgWcFZddxHOLGq82ws/Hk7gSIc+AMlyF74z7Db9i8tFkhXQ2NKldWHV0E3cpXfB4uAjwLU5dhMpFxYnBuZncUUQjEC8oIFk6BzOSbitDJt3k23hzdP+8EV/8NvzzsufynvLYfRN9F30KBpGP0Qvo1fRKLqMSPQh+vfg8KDV+qP1Z+uv1t8r6L2D0ufrqPa0/vkP8GatXA==</latexit>
<latexit
sha1_base64="TAHUg2/62W1xFnpGANmia91W+38=">AAAKQXicjVZfb9s2EJe7f7H3L92wp71wswOkhePZbbEN2AI0SAf0YQO8oGkLWEZAUbRNmCI5kmqcsPpM+xD7FPsIexv2upcdJTmWZBuYEkPnu9/9jnc+HhkpzowdDv9s3Xvn3ffe/+Cg3fnwo48/+fTw/mcvjUw1oZdEcqlfR9hQzgS9tMxy+lppipOI01fR8tzbX72h2jApXtgbRacJngs2YwRbUF0d/h5yOUe/XblQLVh2HCbYLqKZu81QmLAYrb+vsgfoFOVYdRxGksfmJoGXC6kyjEsB9pPCHnI6s29RGFNbyMconGlMIALWlmGONkGyinIXKaxCs/nCPijfbztXh93hYJg/aFsYlUI3KJ/x1f2D2zCWJE2osIRjYyajobJT58MSTrNOmBqqMFniOZ2AKHBCzdTllc3QEWhiNJMaPsKiXFv1cDgxftGA9EmZps0r99nsIrkzdUJBr4lMEixiF9oFfY75XGrmMS7EGkeMuI0q69TJ1gZGst2GusMktbPvp44JlVoqSJHmLOXISuSbBMVMU2L5DQiYAAEjiCxgFcRCK9WY4jdMmbJmq6JoEAoSkJpCijecOsUxE1meYal35TtzL0phYiAeNOS0BlNaKmmYN2STUjl144q2BucU6lcB/px/r0GI1JJzrG8qsPM7XWPhMZ0xsSNORb9hebYHDB2XJqoBPtsoGzFBwHpZpyh1G/eLElTvGiZiqpDfUlSr8Kv8z4tZHUZYNnk8dd0RKj26j4p9GsKcsKj7uNhqg4abqPgJaUvuSpj9PLWWtGx5W29Sk0YEr4tR0Scpt0zL62znpuo0tX6DmgYDAK2U3DTQKwMSbSSoFMmc64Vjw3on4/Ms3InCqZUJVpk7A+GXs/Edqg4zee5Y++W7h3mNKpoG1hethIZ3svd52ESCyX/WlDAsV+7k5CQrwQ00YdpPuALc26B7cqslPHANlxvkbmCxBFkNv2ul1fB7GS1mvODbYLYyIdVqykY1d9aU7K3pVkreWuOvVQrtbV8vcRZpGBy+0TRb9RW2MCCF2QaYBVbU9CNgmWuZingHhmAO4/toj2tMYXblh7bp5ys1fe9QyoOEWtyfMdtfj0Ym5p7MMwGf6xy1lWTCfjPI5wwc5q74ffoo1vi6j5gQVCNI9BRO0yckATLO+0LGFE49AceENacODp4jP7HWpG3P8f8pt9m2q1Bbf7Xyc43hikJWzeZYwl51vaXfs6jx215kk9EUtrSfA1Asd5F1R72mf34ZiZIcGlq6ss5H9dPEXWbOWxz8C3BtHruRlEuLIwPnZ35FEIxAvqCBYugU9om/rYyad5Nt4eWjwejbwfDXJ92nP5b3loPgy+Dr4DgYBd8FT4PnwTi4DEjri9YPrWetn9p/tP9q/93+p4Dea5U+nwe1p/3vfy12xXg=</latexit>
<latexit
sha1_base64="VzUGy9FfqAlQlAoVUr5S5T3pNsQ=">AAAKx3icnVbNbuM2EJa3f4n6s5v22AtbO0WxcFx7N2h7aIAN0sMWaAE32OwuYBkBRdE2IYokSCpxQujQ5+jT9No+Qd+mQ1mOJdkGFpUteDzzzTecITlkrDgzdjj8t/Povfc/+PCjg8Pw408+/ezxk6PPXxuZa0KviORSv42xoZwJemWZ5fSt0hRnMadv4vTC29/cUG2YFK/snaLTDM8FmzGCLaiujzqn0Uxj4iKFtWWYoyjDdhHP3H1R1JSx5Im5y+DHRVQZxqUoCnQWRjGdM+EUOGm2LMIW2f31qM5SeYISfYMikkhrvPBOPmmBoiiMbtZOyVpYabytzZP+j9g7fVJIjIpkk+b1k+5wMCwftC2MKqEbVM/4+ujgPkokyTMqLOHYmMloqOzU+UiEU+DPDVWYpHhOJyAKnFEzdeX8FugYNAmaSQ2vsKjU1j0czoyfHUD62TNtm1fus9lF9mAKI0FvicwyDMlGdkFfYj6XmnmMi7DGMSNuoyrCJtnawEix29B0mOR29uPUMaFySwVZpTnLObIS+aWKEqYpsfwOBEyAgBFEFjAKYmFBN5iSG6ZMVbPlqmgQChKQmkKKd5w6xTETRZlhpXfVb+FeVcLEQDzYFtMGTGmppGHeUEwq5dSNa9oGnFOoXw34a/m/ASFSS86xvqvBLh50rYEndMbEjjg1/Ybl5z1gWHF5plrg842yFRMErNMmRaXbuF9WoOaqYSKhCvldRLWKvio/XiyaMMKKyfOp645Q5dF9hiJOZzaCbmVR93mk2XxhBy03UfMT0lbctTD7eRpL0rL0vrlITR4TvC5GTZ/l3DItb4udmypsa/0GNS0GAFopuWmhlwYk2kpQKVI414vGhvVOxhdFtBOFcyszrAp3DsJv5+MHVBNmytyx9sN3T8sa1TQtrC9aBY0eZO/ztI0Ek3/XlHAqLN3JyUlRgVtowrTvcCtwb4Puya0l4YFruNwgdwNXQ5D18LtGWg+/l9Fixld8G8xWJqReTdmq5s6akr013UrJWxv8jUqhvcvXS5zFGhqHW51OfYUtNEhhtgFmgRU1/RhY5lrmItmBIZhD+z7e45pQ6F3l1cH0y5Gavneo5EFGLe7PmO2vWyMTc0/mmYDPhceHSjJhvxuUfQadIbeanz5KNL7tIyYE1QgSPYPT9JRkQMZ5X8iEwqkn4Jiw5szBwXPsO9aa9NBzvDvlNtt2FRrjr1d+rrFaMLJsL44U9qrrpX7PotbcXhaT0RS2tO8DUCx3WXRHvbZ/eeuKsxIaWbq0zkf13cRdFc5bHHwFuLaP3VjK1OLYwPlZXhEEI5AvaKAYOod94m8ro/bdZFt4/Www+n4w/P20++Kn6t5yEHwZfB18G4yCH4IXwctgHFwFpPNn56/O351/wl9CGd6EyxX0Uafy+SJoPOEf/wHSUfcL</latexit>
<latexit
@✏
= @ ..
@z B .
@z
@✏1
@✏1
0 @z1

k
..
This requires us to compute the Jacobian

···
···
.
f✓, (z) = log p✓ (x, z) − log q (z | x)




Reparameterization Trick

@✏k
@✏k

@zk
. A
.. C
@z1 1
log qφ (z | x) = log p(✏) − log det

for which we need to use “simple” functions g where


@✏ 
@z 
✓ ◆
r Eqφ (z|x) [f (z)] = r Ep(✏) [f (z)] = Ep(✏) [r f (z)]

.
Note that we need to use change of measure formula to write q in terms of epsilon:

You might also like