DRMBackend: Carry pending mode to composite path when direct scanout fails

[nfraprado]

CDRMBackend::Present first attempts to do direct scanout and if that fails it falls back to full compositing. In the case that direct scanout configuration fails, in drm_prepare(), drm_rollback() is called, which overwrites drm->pending with drm->current, discarding any pending mode if there was one, which will now not be applied as part of the full composite commit.

In other words, if for some reason no working direct scanout configuration can be found, no display mode changes will be applied.

Fix this by saving pending.mode_id before the direct scanout drm_prepare() attempt and restoring it on failure when needs_modeset is set.

This was observed on the SteamDeck OLED when displaying an application with resolution 3840x2160 to the internal display, which requires a greater scaling ratio than what the kernel commit abc0ad6d0844 ("drm/amd/display: Limit Scaling Ratio on DCN3.01") [1] allows, resulting in the atomic check failing for the direct scanout path. Under these circumstances, the application is always composited, but only when the "Force Composite" setting is enabled do changes in the panel frequency take effect, since only in that case is direct scanout not even attempted.

[1] torvalds/linux@abc0ad6

Assisted-by: Claude:claude-sonnet-4.6

[oSoMoN]

Wouldn't it be cleaner to encapsulate that logic in drm_prepare() (because drm_rollback() is called from there)?

[nfraprado]

@oSoMoN The problem with that is that there are 3 calls to drm_prepare() in CDRMBackend::Present. We want to carry the pending mode from the first call to the second (compositing), but we do not want to carry it from the second to the third (reattempting with current mode), as the comment there explains. For that one we do want drm_rollback() to have overwritten g_DRM->pending with g_DRM->current so the current mode gets reapplied.

[oSoMoN]

That makes sense, and I now realize that this was already quite clearly explained in the description. Not actually tested, but the change looks sensible to me.