1 Million Tiny Experts in an AI? Fine-Grained MoE Explained

To try everything Brilliant has to offer—free—for a full 30 days, visit https://brilliant.org/bycloud/ . You’ll also get 20% off an annual premium subscription. Mixture of Experts explained, well, re-explained. We are in the Fine-Grain era of Mixture of Experts and it's about to get even more interesting as we further scale it up. This video was sponsored by Brilliant Check out my newsletter: https://mail.bycloud.ai Special thanks to LDJ for helping me with this video Mixtral 8x7B Paper [Paper] https://arxiv.org/abs/2401.04088 Sparse MoE (2017) [Paper] https://arxiv.org/abs/1701.06538 Adaptive Mixtures of Local Experts (1991) [Paper] https://direct.mit.edu/neco/article-abstract/3/1/79/5560/Adaptive-Mixtures-of-Local-Experts?redirectedFrom=fulltext Gshard [Paper] https://arxiv.org/pdf/2006.16668 Branch-Train Mix [Paper] https://arxiv.org/pdf/2403.07816 DeepSeek-MoE [Paper] https://arxiv.org/abs/2401.06066 MoWE (from the meme at 7:51) [Paper] https://arxiv.org/abs/2311.10768 Mixture of A Million Experts [Paper] https://web3.arxiv.org/abs/2407.04153 This video is supported by the kind Patrons & YouTube Members: 🙏Andrew Lescelius, alex j, Chris LeDoux, Alex Maurice, Miguilim, Deagan, FiFaŁ, Robert Zawiasa, Daddy Wen, Tony Jimenez, Panther Modern, Jake Disco, Demilson Quintao, Shuhong Chen, Hongbo Men, happi nyuu nyaa, Carol Lo, Mose Sakashita, Miguel, Bandera, Gennaro Schiano, gunwoo, Ravid Freedman, Mert Seftali, Mrityunjay, Richárd Nagyfi, Timo Steiner, Henrik G Sundt, projectAnthony, Brigham Hall, Kyle Hudson, Kalila, Jef Come, Jvari Williams, Tien Tien, BIll Mangrum, owned, Janne Kytölä, SO, Richárd Nagyfi, Hector, Drexon, Claxvii 177th, Inferencer, Michael Brenner, Akkusativ, Oleg Wock, FantomBloth [Discord] https://discord.gg/NhJZGtH [Twitter] https://twitter.com/bycloudai [Patreon] https://www.patreon.com/bycloud [Music] massobeats - daydream [Profile & Banner Art] https://twitter.com/pygm7 [Video Editor] @Askejm