NoLoCo: training large models with no all-reduce
This is an academic paper describing NoLoCo, a novel optimisation method for distributed training that replaces the global synchronisation step with a gossip method, enabling training
This is an academic paper that finds benefits to heterogeneity (different model sizes and number of training steps) when training embarrassingly-parallel ensembles of expert models. We find that diverse models, trained for varying
This is an academic paper for efficient communication in pipeline parallel training. It introduces an optimal scheduling algorithm that maximises performance and fault tolerance whilst minimising convergence impact from layer skips. It reduces
This is open source code (MIT Licence) for peer-to-peer nodes that perform collaborative reinforcement learning over the internet, accessible by anyone on consumer or datacentre hardware.
We’ve long believed that the future
This is an academic paper describing Verde, a verification protocol for machine learning programs, as well as the underlying Reproducible Operators (RepOps) system that enables it. RepOps is a library that ensures bitwise
