In this talk, we will go through the PyTorch advancements for Large Language Models (LLMs), developments that enhance every aspects of the LLM lifecycle. This includes our newest features/tools to enable large scale training, memory efficient fine-tuning, and on device LLM capabilities.