[Feature]: Integrate sm8x/sm11x support into #40991

[ehfd]

🚀 The feature, motivation and pitch

vllm-project#38476 integrates support for SM80 (Ampere)/SM110 (Jetson Thor) as well as SM12x. It is very trivial to add SM80/SM110 support in vllm-project#40991 as well.

Ultimately, whose PR to be merged would be based on which implementation being most performant, but I would like to make sure that all possible architectures are supported in either PR.

Thank you very much for your PR, too! @jasl.

Alternatives

No response

Additional context

No response

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[ehfd]

Looks like AppMana@641e646 already does it. Just need to expand to sm110.

[jasl]

Sorry for the late response.

The reviewer think my PR too big. I found we can't reuse exising kernels, and no 3rd party kernels suite our need.

So, I think the best stratagy is build the minimum kernels first, and then, convincing the vLLM team to allow to expand to SM8x and SM11x

I'm also owned Jetson Thor, but 128GB memory isn't enough for DSv4, may I ask how can you made it work on Thor?

[ehfd]

I'm also owned Jetson Thor, but 128GB memory isn't enough for DSv4, may I ask how can you made it work on Thor?

Multi-node, I guess. It's the same memory as a DGX Spark, where they are connected with switches.

We can continue with vllm-project#38476, I guess... This is a much smaller and compact PR.