We propose IR3DE, a lightweight router that directs prompts to the most suitable domain experts based on token embeddings. IR3DE enables the addition or removal of experts with minimal updates, avoiding the need to rebuild the router. As a result, it is highly suitable for decentralised settings.

Introducing DEI (Diversity in Evolutionary Inference), a distributed Quality-Diversity search framework that uses heterogeneous LLMs as complementary mutation operators. This allows nodes running different LLMs to collaborate and reach a solution which outperforms solutions by an individual LLMs.

Our method, F-TIS, enables diverse models to collaborate in decentralised GRPO by combining truncated importance sampling with filtering. Across model-size, expertise, and PEFT heterogeneity, F-TIS matches on-policy convergence and, in some cases, improves out-of-distribution reasoning.

Most works on adversarial robustness in distributed training focus on data parallelism, e.g., poisoned gradients, malicious clients, or aggregation attacks. Pipeline parallelism has a different attack surface. Here, the model itself is partitioned across nodes.

Dynamic Upcycling MoE (DUME) cleverly reuses dense experts trained on different domains to create a unified MoE multidomain expert model. DUME retains the knowledge of the original dense experts without any additional training, offering a cost-effective, scalable solution with no compromises.