Eros Fanì

We propose IR3DE, a lightweight router that directs prompts to the most suitable domain experts based on token embeddings. IR3DE enables the addition or removal of experts with minimal updates, avoiding the need to rebuild the router. As a result, it is highly suitable for decentralised settings.

Dynamic Upcycling MoE (DUME) cleverly reuses dense experts trained on different domains to create a unified MoE multidomain expert model. DUME retains the knowledge of the original dense experts without any additional training, offering a cost-effective, scalable solution with no compromises.