Port NVIDIA Cosmos Transfer support to ue5-dev

[youtalk]

Current state

NVIDIA Cosmos Transfer integration currently exists only on ue4-dev. It was added there in #9122 and extended in #9297, but it was never brought forward to the UE5 line. As a result, ue5-dev users cannot generate Cosmos Transfer control inputs (AOV frames + HD-map control videos) or drive a Cosmos Transfer server to produce hyper-realistic style-transfer variations of CARLA synthetic data. The ue4-dev implementation cannot be cherry-picked directly: it relies on the UE4 lane/boundary extraction path and a UE4-era tagged-component / stencil flow that no longer exist on ue5-dev.

Goal

Bring full Cosmos Transfer support to ue5-dev with the same end-user workflow as ue4-dev — replay a log in CARLA, capture AOV frames, export the HD-map geometry to Cosmos-compatible JSON, render the HD-map control video, and call a Cosmos Transfer server to restyle the run — while keeping the exported JSON byte-compatible with the existing Cosmos Transfer / RDS-HQ tooling so downstream consumers and the cosmos-drive-dreams utilities work unchanged.

Approach

• Re-implement lane/boundary extraction on carla::road::Map instead of the UE4-specific path, so the HD-map geometry (lane lines, road boundaries, crosswalks, wait lines, road markings, traffic signs) is derived from the OpenDRIVE road model available on ue5-dev.
• Port the Stencil subsystem to LibCarla (Stencil, RoadInfoStencil, and the Map / MapBuilder / object-parser support) so stencil/road-info data is available client-side without the removed UE4 tagged-component machinery.
• Keep the exported JSON byte-compatible with the existing Cosmos Transfer format via dedicated export_cosmos_* exporters and RPCs, validated against the reference artifacts.
• Ship the work in two phases. Phase 1 delivers the geometry/exporter/pipeline code plus docs and an on-PC capture (AOV frames, HD-map control video, optional server-side restyle), gated on visual verification. Phase 2 covers the remaining polish and the control-sensor work that builds on the Phase 1 foundation.

Implementation plan: four stacked PRs (against ue5-dev)

The implementation will be submitted as four stacked PRs, bottom-up, so each layer can be reviewed independently:

1. Stencil subsystem → LibCarla: port Stencil, RoadInfoStencil, and the Map / MapBuilder / object-parser support so stencil/road-info data is available client-side without the removed UE4 tagged-component machinery.
2. export_cosmos_* exporters + RPCs: crosswalks, road boundaries, lane lines, traffic signs, wait lines, and road markings, with the lane/boundary geometry re-derived from carla::road::Map and the exported JSON kept byte-compatible with the existing Cosmos Transfer / RDS-HQ format.
3. Cosmos Transfer Python pipeline + docs: the branch-independent client/server pipeline (carla_cosmos_gen.py, cosmos_client.py, conda environments, Dockerfiles, example_data), the cosmos-drive-dreams submodule, the ported docs (Docs/nvidia_cosmos_transfer.md + images, mkdocs nav), and the sensor.camera.cosmos_visualization smoke test.
4. Phase 2 follow-up: the in-engine control sensor and remaining polish that builds on the first three PRs.

[JArmandoAnaya]

Hello @youtalk, thank you for this proposal. I agree with this change. I was also thinking of the porting of Nvidia Cosmos and NuRec from UE4 to UE5 since this is one of the powerful features existing in Carla.

Furthermore, I saw you already made the PRs to achieve this posting. Did you have changes to test locally the integration with the NVIDIA models, and if so, I'm curious about how you feel about the performance in Carla UE5?

Thanks again for this work.

[youtalk]

@JArmandoAnaya NVIDIA Cosmos Transfer is extremely resource-intensive, so I ran it on an NVIDIA H100 GPU via NVIDIA Brev's cloud server. The results are shown in the video below.
#9798 (comment)

One drawback of style transfer technology is that the final artistic output is almost entirely independent of the original video's quality. Even if you use high-quality footage rendered in Unreal Engine 5, it won't necessarily result in a cleaner or better-looking style transfer video.

[JArmandoAnaya]

Awesome! It looks really great. I hope this can be merged soon. I'll compile and run this on the Cloud when I have the opportunity to validate it on my side.

[JArmandoAnaya]

@youtalk Did you try running it on NVIDIA Brev's cloud server using the Lambda option, as the current Carla documentation says? because Iḿ tryoing to run it, but that option looks unavailable. It says that I can use Hyperstack. Did you use it? I wonder if you know if Hyperstack could also work for this purpose.

[youtalk]

I used the Hyperstack with a single H100. The throughput is slower but it works enough with reasonable fees.

[youtalk]

The disk size of the root directory is very small, so you need to use /ephemeral disk. Please keep that in mind.

[JArmandoAnaya]

Thank you, I will try under that setup.

[M-Colley]

Very very cool work! I look forward to testing it :)

Does this already also support Cosmos 3? (https://github.com/NVIDIA/cosmos/releases/tag/Cosmos3)

[youtalk]

@M-Colley It must be great! I'll test it.

[youtalk]

@M-Colley @JArmandoAnaya Testing for the Cosmos Transfer 2.5 is now complete. The output quality is better than 1.0. I plan to submit this PR upstream as soon as the four pending PRs for Cosmos Transfer 1 are merged.
youtalk#12

Since Cosmos 3 is an omni-model supporting a wide range of functions, it isn't very compatible with our existing scripts. I intend to try adjusting it again once 2.5 has been merged.

[JArmandoAnaya]

@youtalk great work!! That sounds wonderful. An upgrade to Cosmos Transfer 2.5 is a big progress. Thank you for your hard work.