Traceml

TraceML is an open source diagnostics tool for PyTorch training that shows where step time actually goes: data loading, compute, transfers and memory. It is the engine behind TraceOpt, a shared training performance platform the vendor says it is building next. It is aimed at machine learning engineers who lose GPU hours to slow or unstable training runs. A user installs it from PyPI, adds one import, one init call and a context manager around the existing training step, then launches the script with the traceml run command. It profiles the step and saves a run summary that can be inspected or compared with later runs. It detects common bottlenecks such as slow input loading, low GPU utilization, rank stragglers, residual heavy steps, memory imbalance and memory creep. The live view reports median step time, the slowest worker, the gap between workers and GPU memory use. It supports single node multi GPU DDP and FSDP, and multi node DDP in summary mode, and lists support for Lightning, Hugging Face, Ray Train, Slurm, DeepSpeed and Megatron LM as supported or planned. The install needs no Docker, account or hosted backend. The vendor describes overhead as low from early internal runs, and says actual impact depends on workload and hardware. The TraceOpt team workflows for regression comparison and shared diagnostics are described as work with design partners, and no prices are published.