Scale-out network
Beginner
The network that joins the servers across a whole building. It is slower than the links inside a team, but it reaches thousands of machines.
Novice
The datacenter network (InfiniBand or Ethernet, through layers of switches) that joins many servers or scale-up domains. It reaches far more chips but gives each one less bandwidth and more latency than the scale-up fabric.
Expert
NIC-based, message-oriented (RDMA) networks built as multi-tier Clos or similar topologies. Per-accelerator bandwidth is typically several times lower than scale-up, and latency includes NIC and multi-hop switch traversal, so only parallelism that tolerates that, such as data and pipeline parallelism, is placed across it.