The xpu_timer fails to generate normal timeline information

[dbfancier]

Describe the bug
Hi, after building the xpu_timer, the relevant lib could not be found, and needed to be packaged and installed to /usr/local/lib and /usr/local/bin. The generated matmul_dp8.bin after all this was done seemed to be problematic.

Using xpu_timer_gen_tracing_timeline also seems to produce svg that does not properly reflect timeline information. Are there more documentation or communication group to discuss these issues?

"Stack count is low (8). Did something go wrong?" is displayed after the calltrace is generated.

To Reproduce

Build according to the xpu_timer documentation

Logs or Screenshots

Expected behavior
generate normal timeline info

APP Info :

• xpu_timer: 0.4.0-rc
• Torch 2.4.0
• cuda 12.5
• NCCL 2.22.3

HARDWARE Info :

• Device: GPU A800

[cos120]

@dbfancier There should be 3 files after you running xpu_timer_gen_tracing_timeline

1. matmul_*.bin is for replay back performance for matmul, it is not a timeline file of perfetto
2. trace_*.bin is the timeline file for perfetto
3. tracing_kernel_stack.svg, this file is for callstack of your cuda api/cublas api

Could you show the files that you generated?

And one more thing, tracing nccl which greater than 2.21.5 should not work, since nccl change cuda api to driver api, we are fixing it.

[dbfancier]

@dbfancier There should be 3 files after you running xpu_timer_gen_tracing_timeline

1. matmul_*.bin is for replay back performance for matmul, it is not a timeline file of perfetto
2. trace_*.bin is the timeline file for perfetto
3. tracing_kernel_stack.svg, this file is for callstack of your cuda api/cublas api

Could you show the files that you generated?

And one more thing, tracing nccl which greater than 2.21.5 should not work, since nccl change cuda api to driver api, we are fixing it.

Thanks for your reply, I will replace the nccl version and try profiling again, and I will post the generated content to discuss with you later.

[lvdunlin]

1. matmul_*.bin

What software is needed to open matmul_.bin files? @cos120

[dbfancier]

1. matmul_*.bin

What software is needed to open matmul_.bin files? @cos120

Thank you for your reply.

I guess I may have opened the wrong file, in fact I don't know the specific function of matmul*.bin, so I thought it could be opened and read like trace_*.bin.

In fact, the other two issues are more concerning to me.

1. It seems that the relevant so file is not automatically installed correctly during installation

2. The generated flame information seems not to be too accurate, do you see anything strange in the flame I generated? It seems that its upper stage and the lower stage are always the same width. And if there are N functions above it, then they are equally divided.

I haven't had time to generate a flame with another training job recently, so if I have more information, I will post it as soon as possible and ask for your help.

[lvdunlin]

And one more thing, tracing nccl which greater than 2.21.5 should not work, since nccl change cuda api to driver api, we are fixing it.

@cos120 tracing nccl which greater than 2.21.5，when can it be released?

[Edward-liang]

And one more thing, tracing nccl which greater than 2.21.5 should not work, since nccl change cuda api to driver api, we are fixing it.

@cos120 tracing nccl which greater than 2.21.5，when can it be released?

I am wondering the same question, are there any detailed plan?

[cos120]

@lvdunlin @Edward-liang We have another design and it will not depend on version of nccl.

[cos120]

1. matmul_*.bin

What software is needed to open matmul_.bin files? @cos120

matmul.bin file is just the param of cublas/cublasLt, it is using for replay performance of matmul, this is design for excluding performance downgrade from overlapping with other gpu kernels(nccl).

[cos120]

1. matmul_*.bin

What software is needed to open matmul_.bin files? @cos120

Thank you for your reply.

I guess I may have opened the wrong file, in fact I don't know the specific function of matmul*.bin, so I thought it could be opened and read like trace_*.bin.

In fact, the other two issues are more concerning to me.

1. It seems that the relevant so file is not automatically installed correctly during installation
2. The generated flame information seems not to be too accurate, do you see anything strange in the flame I generated? It seems that its upper stage and the lower stage are always the same width. And if there are N functions above it, then they are equally divided.

I haven't had time to generate a flame with another training job recently, so if I have more information, I will post it as soon as possible and ask for your help.

1. About the shared object files, there are two kinds of files, you can find log at /tmp/xpu_hook.log.{rank}

• for parsing param of nccl/flash attn, thats we can get the workload of those kernel
• for tracing python host call

2. The flamegraph file is just showing callstack of matmul or nccl, it not represents real cost of each kernel.