Distributed training (multi-node) of a Transformer model

Messaging and state layer for distributed serverless applications

Summary of call graphs and data structures of NVIDIA Collective Communication Library (NCCL)

Blink+: Increase GPU group bandwidth by utilizing across tenant NVLink.

TileXR (eXtreme Rendezvous for Asynchronous Tile Communication) is a data-centric asynchronous communication runtime for Huawei Ascend NPUs.

collectives library for upc++

Audit GPU cluster communication schedules from NCCL logs. Zero dependencies. CI-ready.

Interactive web visualization for understanding collective communication algorithms (as used in NCCL, RCCL, MPI). Learn how AllReduce, Broadcast, Reduce, AllGather and more work step by step.

From-scratch Ring-AllReduce on CPU (C++20) and GPU (CUDA, multi-device): real multi-GPU benchmarks, Nsight profiling, and a bandwidth-optimal ring implementation built without NCCL.

Research prototype investigating adaptive collective communication optimization for MPI workloads using runtime performance feedback.

A deterministic C++20 digital twin for AI training clusters—model workloads, networks, scheduling, congestion, and failures through reproducible experiments.

AllReduce/AllGather scaling in ASTRA-sim across torus vs switch topologies on the analytical + ns-3 backends — latency-bound vs bandwidth-bound over message size and node count. Reproducible Docker/Chakra harness + write-up.

This repository contains simple programs of MPI_Bcast, MPI_Reduce, MPI_Scatter and MPI_Gather. Download the repository and test your self.

Modelling of MPI collective operations latencies: Broadcast and Reduce operations. UniTS, SDIC, 2023-2024

Simple quick test to benchmark your pytorch + nccl/ncclx setup

A reduction algorithm for MPI using only peer to peer communication

Benchmark suite for MPI collectives (Bcast, Alltoall, Allreduce) with from-scratch ring and recursive-doubling allreduce implementations. Fitted Hockney cost models, measured algorithm crossovers, and LaTeX reports generated entirely from the data. C++17, Open MPI, Python analysis pipeline.

Summary of call graphs and data structures of collective communication plugin in NVIDIA TensorRT-LLM

HPC course practice assignments for parallel-programming

Analytical simulator for collective communication on ring, fat tree, and dragonfly topologies: graph based hop costs, bandwidth efficiency, and closed form algorithm crossover up to 1024 nodes. Reproducible sweeps in Python (networkx), with a compiled LaTeX report.