Any plans for PCIe / RDMA metrics?

[Peng-Xu]

Hey, really like this agent — thanks!

Btw, the per-pid GPU utilization via NVML (gpu/gpu.go) is actually
nicer than dcgm-exporter for shared GPU nodes, since dcgm only gives
device-level numbers. Nice touch.

Quick roadmap question — any plans for:

1. PCIe link speed/width + AER errors (from sysfs)?
2. RDMA / InfiniBand counters? TCP tracing misses NCCL traffic.
3. GPU SM occupancy / NVLink / MIG support?

Building a GPU cluster observability stack and would love to use this
as the main node agent. Thanks!

[def]

@Peng-Xu we’d love to add these metrics too, but we don’t yet have a good feel for real-world bottlenecks or access to hardware to see them in action.

For example, we could add PCIe bandwidth metrics, but to make them actionable we’d also need things like GPU idle time, memcpy stats, and topology (GPU-CPU-NIC) to understand if it’s actually a bottleneck.

[Peng-Xu]

Yeah makes sense — metrics alone aren’t super useful without context.
What I’m aiming for is more like full-stack observability, not just more metrics.
In practice, GPU cluster issues usually span multiple layers, not just one — from PyTorch/NCCL all the way down to PCIe, network, GPU, etc.
So the idea is to correlate signals across layers (GPU idle, NCCL, topology, fabric stats, etc) and actually figure out where the bottleneck is.
Basically, less about collecting metrics, more about understanding what’s actually going wrong.

[Peng-Xu]

think there’s a bit of over-coupling here.
The GPU collector is tied into the main agent process, which makes the deployment less flexible.
Ideally, this could be split out — similar to dcgm-exporter — so GPU-specific dependencies don’t impact non-GPU nodes.
A modular or sidecar-based approach might make this cleaner and easier to scale.

Of course, this is all just part of bouncing around ideas on architecture.