Llama 4’s pre-training scale is growing exponentially, with 100K GPUs used, a 6x increase from its predecessor. Initializing training takes longer, and failure probability increases with larger scale. Training throughput aka Effective Training time degrades significantly as a result. To address these challenges, researchers are experimenting in parallel for faster initialization of large scale jobs, and fault-tolerant paradigms.