[richardssam]

Pull Request Decode H.264, HEVC and ProRes with VideoToolbox on MacOs.

Summarize your change.

MacOS has a terrific library videotoolbox that offers hardware decompression of H264 HEVC, Prores and AAC, including licenceses for them. This PR enables the ffmpeg videotoolbox decoders where available. libavformat still does demuxing, seeking, timecode and audio, so every container that works today still works.

The new backend lives inside FFMpegStream and follows avcodec's send/receive/flush behaviour, so FFMpegDecoder, thumbnails and caching are unchanged. Frames stay in native YCbCr, with no RGB conversion by the OS. They're repacked into planar formats the existing YUV shader already reads, so the shader doesn't change.

A new preference, /plugin/media_reader/FFMPEG/videotoolbox, controls it:

• off: always use FFmpeg's decoders, exactly as before
• prefer (default): use VideoToolbox, falling back to FFmpeg's decoder if a VideoToolbox session can't be created
• only: use VideoToolbox, or show an error on the frame

Describe the reason for the change.

• Speed and battery life. On Apple Silicon, H.264, HEVC and ProRes decode on dedicated media hardware. 4K playback uses 5–10× less CPU than FFmpeg's software decoders, and 4K H.264 that stutters with software decoding plays smoothly.
• Apple's own ProRes decoder. FFmpeg's prores decoder is reverse-engineered; VideoToolbox is Apple's reference implementation.
• Groundwork for an official macOS binary without codec patent exposure. Decoding H.264, HEVC and AAC through the OS, where Apple holds the licences, is the first step. A stream VideoToolbox decodes no longer needs FFmpeg to have a decoder for it at all. Removing the patented decoders from the build is left to a follow-up PR. Keeping FFmpeg's full decoder set in this one means every VideoToolbox result can be checked against FFmpeg's.

Describe what you have tested and on which operating system.

macOS 26.6 on an M2 Max, with FFmpeg 8.0.1 from vcpkg.

• Unit Tests:
  • ffmpeg_test has 26 tests, all passing. The test media is generated with src/plugin/media_reader/ffmpeg/test/videotoolbox_media/generate.sh but the media is not checked in. Tests whose clips haven't been generated are skipped.
  • To run ffmpeg_test locally, put vcpkg's lib/manual-link on DYLD_LIBRARY_PATH so gtest_main loads.
• Matching ffmpeg's output:
  • H.264 and HEVC (8/10-bit in 4:2:0, 4:2:2 and 4:4:4, plus BT.2020/PQ) are bit-exact against FFmpeg's decoder, including frames reached by jumping backwards and forwards.
  • 12-bit HEVC is within ±1 code value, from rounding 16 bits down to 12.
  • ProRes, all six profiles, is within 4 code values, with alpha identical.
  • Images rendered through the app's colour pipeline are pixel-identical for H.264 and HEVC, and differ by at most 0.008 for ProRes.
• In the app:
  • Playback, thumbnails in the media list, pixel probe values, AAC sync and scrubbing all checked.
  • The preference appears in Preferences, and changing it at runtime switches decoders.
  • The log shows one line per source naming the decoder used.
• Not tested: Linux and Windows, where all the new code is compiled out except the platform-neutral repack loops. Also not tested: a base M1 (no ProRes engine) or an Intel Mac. VideoToolbox decodes in software there, so the speed gain will be smaller.

Add a list of changes, and note any that might need special attention during the review.

• New: vt_decoder.{hpp,cpp}, the VideoToolbox backend. It's built only on Apple, and uses the C APIs, so there's no Objective-C++. Frames are reordered into presentation order using the highest decode timestamp submitted, so there's no SPS parsing.
• New: vt_repack.{hpp,cpp}, platform-neutral, vectorised loops that unpack VideoToolbox's two-plane formats into planar ImageBuffers. Each takes 0.4–2.2 ms per 4K frame. They replaced swscale, which took 7.7–9.6 ms.
• ⚠️ FFMpegStream constructor: VideoToolbox is tried before avcodec_find_decoder(), and codec_ can now be null for video streams. Uses of codec_->id became codecpar->codec_id. The software setup moved into open_software_video_decoder() without changing behaviour.
• ⚠️ Default is prefer: macOS users get VideoToolbox without doing anything. ProRes values differ from FFmpeg's prores decoder by up to 4 code values; Apple's decoder is the reference.
• ⚠️ cmake/macros.cmake: gtest targets now get the same __apple__ define as other targets. Without it, ffmpeg_test didn't compile on macOS, on develop too, because the headers rely on that define to undefine the nil macro from MacTypes.h.
• Output formats: ProRes 4444 uses s4as. VideoToolbox's conversion to y416 replaces real chroma with neutral values over about 12% of the frame. 12-bit 4:2:0 HEVC stays on FFmpeg, because VideoToolbox's chroma for it is inaccurate.
• FFMpegMediaReader: on Apple, AAC prefers aac_at. On a preference change, the reader drops its last decoded frame and its decoders. A new videotoolbox_unavailable_error is always shown on the frame.
• Test-only hook: the environment variable XSTUDIO_VIDEOTOOLBOX_FORCE_FAILURE makes VideoToolbox session creation fail.
• Existing issues found while testing, not fixed here:
  • Raw Annex-B .h264 files have no duration and show black.
  • The thumbnail service stalls when a reader returns no thumbnail, as it does for a raw .h264.
  • The frame just before an open-GOP keyframe can't be reached after a backward jump with FFmpeg's decoder.
  • Python API: MediaCache.erase() is a no-op, and OffscreenViewport.render_media_frame() uses an undefined variable.

Co-Authored-By: Claude Opus 5.5 noreply@anthropic.com