This is an academic paper that finds benefits to heterogeneity (different model sizes and number of training steps) when training embarrassingly-parallel ensembles of expert models.

This is an academic paper for efficient communication in pipeline parallel training. It introduces an optimal scheduling algorithm that maximises performance and fault tolerance whilst minimising convergence impact from layer skips.

This is an academic paper describing Verde, a verification protocol for machine learning programs, as well as the underlying Reproducible Operators (RepOps) system that enables it.