[Glean] NVIDIA NCCL Introduction

Published: June 26, 2024 by SingularityKChen (Last updated: June 27, 2024 )

• Categories:
• Glean 53

• Tags:
• NCCL 1
• NVIDIA 5

• NCCL
  • Introduction
  • Reference

NCCL

NVIDIA Collective Communication Library (NCCL) is optimized primitives for inter-GPU communication.¹

Introduction

The NVIDIA Collective Communications Library (NCCL, pronounced “Nickel”) is a library providing inter-GPU communication primitives that are topology-aware and can be easily integrated into applications.²

NCCL implements both collective communication and point-to-point send/receive primitives. It is not a full-blown parallel programming framework; rather, it is a library focused on accelerating inter-GPU communication.

NCCL provides the following collective communication primitives :

• AllReduce
• Broadcast
• Reduce
• AllGather
• ReduceScatter

Additionally, it allows for point-to-point send/receive communication which allows for scatter, gather, or all-to-all operations.

Reference

• NCCL: High-Speed Inter-GPU Communication for Large-Scale Training
• Massively Scale Your Deep Learning Training with NCCL 2.4

1. GitHub - NCCL ↩

2. NVIDIA Collective Communication Library (NCCL) Documentation ↩