Training incredibly large AI models, such as the latest large language models, presents significant challenges. These training runs can span weeks or even months, utilizing thousands of specialized processors like GPUs or TPUs distributed across a cluster. During such extended, complex operations, system failures are an unfortunate reality. Hardware issues, software glitches, network interruptions, or power outages can halt the training process. Without proper safeguards, a failure could mean losing hours, days, or even weeks of computational work and valuable progress, requiring a costly restart from a much earlier point.
To mitigate this risk and minimize the impact of failures, a standard practice in large-scale distributed training is checkpointing. This involves periodically saving the complete state of the model – including model weights, optimizer states, learning rate schedules, and any other necessary information – to stable storage. If a failure occurs, training can be resumed from the most recent successful checkpoint, saving considerable time and resources compared to starting over.
However, as AI models grow exponentially in size, the amount of data needed to represent their state also becomes massive. Saving a full checkpoint for a trillion-parameter model can involve writing terabytes of data. Performing this full checkpointing frequently enough to minimize data loss can become a significant bottleneck, slowing down the training process itself due to the time and resources required for saving. This overhead can negate some of the benefits of checkpointing.
This is where an advanced technique called multi-tier checkpointing offers a powerful solution. Instead of treating all state data equally and saving everything to one location at a single frequency, multi-tier checkpointing strategically saves different components of the model state to different storage tiers with varying frequencies. This approach leverages the characteristics of different data types and storage systems to optimize both fault tolerance and training efficiency.
A typical multi-tier system might involve at least two or three distinct tiers:
Fast, Frequent Checkpoints: This tier focuses on saving critical, rapidly changing state information, such as the optimizer state. Optimizer states can be very large but change with every training iteration. Saving this frequently is crucial to minimize the amount of work lost in the event of an immediate failure. These checkpoints are often saved to very fast storage, potentially local NVMe drives or even shared memory on the compute nodes themselves. This tier provides high resilience against common, localized failures. The overhead per checkpoint is kept low because only a subset of the total state is saved.
Less Frequent, Comprehensive Checkpoints: This tier is responsible for saving the full model state, including the large model weights and the corresponding optimizer state from the fast tier. Since weights change more slowly and saving them is more resource-intensive, these comprehensive checkpoints are performed less often than the fast ones – perhaps every few hundred or thousand training iterations. This process is often offloaded or performed asynchronously to avoid blocking the primary training computation. Data for this tier is typically saved to a high-speed, distributed filesystem or dedicated checkpointing storage, providing good performance without excessive cost.
Durable, Long-Term Archival: Periodically, perhaps daily or weekly, a full, robust checkpoint from the comprehensive tier is copied to highly durable long-term storage, such as object storage in the cloud or on-premises. This tier protects against larger-scale disasters, such as the failure of the entire cluster or primary storage system. While retrieval from this tier might be slower, its purpose is long-term reliability and archival for compliance or future research.
By implementing a multi-tier checkpointing strategy, organizations training massive AI models can achieve a better balance between fault tolerance and training speed. It significantly reduces the potential data loss and costly re-training time associated with failures. By optimizing where and how different parts of the model state are saved, the overhead of checkpointing is minimized, leading to more efficient use of computational resources and ultimately faster progress towards training complete, high-performing AI models. This strategic approach is becoming increasingly essential as model sizes continue to grow and training runs become longer and more complex.
Source: https://cloud.google.com/blog/products/ai-machine-learning/using-multi-tier-checkpointing-for-large-ai-training-jobs/
