Coming soon

Software

Chips and networks only matter once software puts them to work. This guide will show how an AI program written by a researcher ends up running on thousands of chips.

This guide will cover the layers between a model and the hardware: the compilers that turn it into chip instructions, hand-tuned kernels, the runtimes that schedule work, and the libraries that move data between chips.

This guide will cover graph compilers and lowering, kernel design for matrix units and memory hierarchies, runtime scheduling, and collective communication libraries, picking up where the Systems guide’s parallelism chapter stops.

Guide in progress

What this guide will cover

  1. ML compilers
  2. Kernels
  3. Runtimes
  4. Collective communication libraries

Until then, the Systems guide’s chapter on why the network looks this way covers how splitting a model across chips shapes the hardware.