It is a distributed computing technique used to split the parameters (weights and biases) and computations of a single neural network layer across multiple GPUs.