fix(video): fall back to SDR instead of failing session when HDR isn't supported

[matix753]

Description

When a client requests HDR (dynamicRange) for HEVC or AV1 but the active capture/encode
device combination can't actually deliver it, make_avcodec_encode_session() fails the
session outright (return nullptr) instead of falling back to SDR the way it already does
for H.264 (encoder.h264[DYNAMIC_RANGE] is hardcoded false, so H.264 never even attempts
HDR).

On Linux, NvFBC capture always uses the cuda::cuda_t encode-device class
(src/platform/linux/cuda.cpp), which only ever produces AV_PIX_FMT_NV12 /
AV_PIX_FMT_YUV444P frames — no P010/10-bit path, regardless of what the encoder itself
supports. The sibling class gl_cuda_vram_t (GL/dmabuf capture path) does support
AV_PIX_FMT_P010LE, so this is a gap specific to the NvFBC+CUDA path, not a hardware
limitation. Confirmed on an NVIDIA Quadro T1000 (Turing): ffmpeg -c:v hevc_nvenc -profile:v main10
encodes 10-bit HEVC fine outside of Sunshine, but inside Sunshine with capture = nvfbc any
client HDR-HEVC request fails with:

Error: cuda::cuda_t doesn't support any format other than AV_PIX_FMT_NV12 and AV_PIX_FMT_YUV444P
Error: Could not open codec [hevc_nvenc]: Function not implemented

Some clients (observed with the LG webOS Moonlight app, mariotaku/moonlight-tv) don't give
up after the failed session — they silently reconnect and request the same HDR-HEVC config
again, forever. Sunshine fails every attempt cleanly (no loop inside Sunshine itself), but
from the outside it looks like a permanent hang.

This PR makes make_avcodec_encode_session() downgrade to SDR in place instead of failing
the session, mirroring what H.264 already does: a local dynamicRange bool starts as
config.dynamicRange, gets set to false (plus encode_device->colorspace corrected to
rec709/8-bit) when the YUV444 or 4:2:0 checks find HDR unsupported, and every later use of
config.dynamicRange in the function reads this local instead. The startup capability probe
(probe_encoders() / test_yuv420_hdr()) is unaffected, since it already reads the resulting
encoder.hevc[DYNAMIC_RANGE] flag and skips advertising HDR when this path degrades it.

Screenshot

N/A — server-side encode path change, no UI.

Issues Fixed or Closed

Roadmap Issues

Type of Change

• feat: New feature (non-breaking change which adds functionality)
• fix: Bug fix (non-breaking change which fixes an issue)
• docs: Documentation only changes
• style: Changes that do not affect the meaning of the code (white-space, formatting, missing semicolons, etc.)
• refactor: Code change that neither fixes a bug nor adds a feature
• perf: Code change that improves performance
• test: Adding missing tests or correcting existing tests
• build: Changes that affect the build system or external dependencies
• ci: Changes to CI configuration files and scripts
• chore: Other changes that don't modify src or test files
• revert: Reverts a previous commit
• BREAKING CHANGE: Introduces a breaking change (can be combined with any type above)

Checklist

• Code follows the style guidelines of this project
• Code has been self-reviewed
• Code has been commented, particularly in hard-to-understand areas
• Code docstring/documentation-blocks for new or existing methods/components have been added or updated
• Unit tests have been added or updated for any new or modified functionality

Manually verified on real hardware (Quadro T1000, NvFBC capture) and a real client (LG webOS
Moonlight): HDR-HEVC request now logs a falling back to SDR warning, the session starts
successfully, and streaming is stable with zero reconnect loop. HDR still works correctly for
capture paths that support it (unaffected by this change).

AI Usage

See our AI usage policy.

• None: No AI tools were used in creating this PR
• Light: AI provided minor assistance (formatting, simple suggestions)
• Moderate: AI helped with code generation or debugging specific parts
• Heavy: AI generated most or all of the code changes

[ReenigneArcher]

Thank you for the PR submission, but it looks like you used AI to create this PR.

Please read and follow our Contributing guidelines and specifically our AI Usage policy.

Additionally, please update the PR to use the correct template. You can find it at https://github.com/LizardByte/.github/blob/master/.github/pull_request_template.md?plain=1

[matix753]

Hi @ReenigneArcher , fair callout, and sorry for the friction.

To be clear about what AI was actually used for here:

• The fix itself in src/video.cpp was written, compiled, and debugged by hand. I traced the bug from a real symptom (infinite reconnect loop on an LG webOS Moonlight client requesting HDR-HEVC) down to cuda::cuda_t in src/platform/linux/cuda.cpp never producing a P010 frame, and verified on real hardware (Quadro T1000) with a real client that the fallback works and HDR still works where it's supported.
• I did use AI for one specific part of testing: a high-frequency stress loop to check packet send/response stability under load, to make sure the fallback didn't introduce a regression under real traffic. That's test tooling, not the shipped code.
• I also used AI to help write the original PR description, which is why it read the way it did. I've rewritten it against your standard template and marked AI Usage as Moderate, which is the honest answer.

Should have disclosed all this and used the template from the start instead of after the fact. Let me know if anything else needs fixing before this gets a real look.

[ReenigneArcher]

Thanks for the clarification. And to clarify on my side, using AI is no problem. I just do not want to have a discussion with LLMs in PR reviews or read scientific reports for PR descriptions (often far longer than the actual changes to the code).

Anyway, the PR seems fine to me. Thanks!

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud