Great article. Some of these intuitions, such as why MoE can fail in the beginning and how the MoE can be equivalent to a dense model with some many params are super useful.
Any insight into methods that could be solved to make the routing problem better? Additionally, as OAI is not really a consumer company, I doubt their decision was made based on a lower inference cost (they just want the best models to find AGI-like things). Thoughts on what that means?
I can't wait to see how these advancements will shape the future of AI. Keep up the fantastic work!
Great article. Some of these intuitions, such as why MoE can fail in the beginning and how the MoE can be equivalent to a dense model with some many params are super useful.
Any insight into methods that could be solved to make the routing problem better? Additionally, as OAI is not really a consumer company, I doubt their decision was made based on a lower inference cost (they just want the best models to find AGI-like things). Thoughts on what that means?