feat(video): decode supported VP9 streams with VideoToolbox

CHANGELOG.md

@@ -12,6 +12,12 @@ the public-API contract.
 
 ### Added
 
+- **VP9 can use VideoToolbox hardware decoding on the software playback host.** WebM streams
+  synthesize the required `vpcC` record from probed parameters. The hardware candidate accepts
+  8-bit profile 0 and 10-bit profile 2 with 4:2:0 chroma, requires a hardware session, and falls
+  back to libavcodec if registration, configuration or session creation fails. Decoder diagnostics
+  report the implementation that actually opened.
+
 - **Remote MPEG-DASH on-demand manifests play (AE#859, thanks @qoli).**
   A static, single-Period MPD whose representations are single files addressed by `SegmentBase`
   failed at open with `Invalid data found when processing input`: the FFmpeg build has no DASH

README.md

@@ -66,7 +66,7 @@ A scannable summary; the depth for each row lives in **[docs/formats.md](docs/fo
 | MPEG-DASH | Remote on-demand manifests (static, one Period, `SegmentBase`, unencrypted): the video and every audio language are laid out as one fragmented MP4, so seeks, audio switching and both playback paths work unchanged. Live and `SegmentTemplate` manifests are refused with a reason |
 | Disc | DVD-Video and Blu-ray ISO (decrypted): selectable titles and chapters, demuxed through the normal path |
 | Video (HW) | H.264, HEVC, HEVC Main10 via VideoToolbox; AV1 Main profile where HW AV1 exists |
-| Video (SW) | AV1 (dav1d) without HW, and AV1 High / Professional (4:4:4, 4:2:2, 12-bit) everywhere, VP9 / VP8, MPEG-4 Part 2 / MPEG-2 / VC-1, QuickTime RLE, the Flash tail (Sorenson Spark, On2 VP6), RealVideo 1-4, H.263, Theora and anything else the FFmpeg build carries a decoder for (software is the default route; only HEVC, H.264 and HW-decodable AV1 go native), H.264 High 4:2:2 / 4:4:4 / 10 and HEVC Rext where VideoToolbox has no HW decoder (Intel Macs, older chips), interlaced H.264 (AVPlayer does not deinterlace; on VOD the declared field order is verified against decoded frames, so progressive-in-interlaced-carriage keeps hardware decode); GPU deinterlace (yadif_videotoolbox, Metal, field-rate by default) with a CPU bwdif fallback |
+| Video (SW) | AV1 (dav1d) without HW, and AV1 High / Professional (4:4:4, 4:2:2, 12-bit) everywhere, VP9 (VideoToolbox hardware when supported, otherwise libavcodec) / VP8, MPEG-4 Part 2 / MPEG-2 / VC-1, QuickTime RLE, the Flash tail (Sorenson Spark, On2 VP6), RealVideo 1-4, H.263, Theora and anything else the FFmpeg build carries a decoder for (software is the default route; only HEVC, H.264 and HW-decodable AV1 go native), H.264 High 4:2:2 / 4:4:4 / 10 and HEVC Rext where VideoToolbox has no HW decoder (Intel Macs, older chips), interlaced H.264 (AVPlayer does not deinterlace; on VOD the declared field order is verified against decoded frames, so progressive-in-interlaced-carriage keeps hardware decode); GPU deinterlace (yadif_videotoolbox, Metal, field-rate by default) with a CPU bwdif fallback |
 | HDR | HDR10, HDR10+ (per-frame ST 2094-40), Dolby Vision (P5, P7 on the native route as single-layer 8.1 (HDR10 base layer on a display without Dolby Vision), or composed dual-layer on Metal with per-frame DV metadata for FEL when `preferDualLayerDolbyVision` is set, the `DualLayerDolbyVision` trait is on and VideoToolbox decodes HEVC in hardware, P8.1, P8.4, AV1 P10.x), HLG (declared, or named by an alternative transfer SEI as broadcast HLG is), HDR Vivid (played on its HLG / PQ base, CUVA metadata reported by `probe`) |
 | Audio | AAC, AC3, EAC3, FLAC, ALAC stream-copy lossless; TrueHD / MLP / DTS / DTS-HD MA / MP3 / MP2 / Opus / Vorbis / LPCM (incl. Blu-ray, G.711) / WMA (Standard, Pro, Lossless, Voice) / Nellymoser / ADPCM-SWF / Speex / RealAudio (Cook, ATRAC3, Sipro, Lossless, 14.4 / 28.8) / AMR-NB / AMR-WB / DVD LPCM bridge to EAC3 5.1 (default) or lossless FLAC |
 | Dolby Atmos | EAC3+JOC stream-copied on every route (HDMI MAT 2.0, AirPods spatial, BT downmix) except the opt-in Profile 7 FEL dual-layer route (`preferDualLayerDolbyVision`, `DualLayerDolbyVision` trait), which runs on the software host and plays an E-AC-3 JOC track as its 5.1 / 7.1 bed without the objects unless `LoadOptions.objectAudioRendering` is set (then the objects are kept: compressed JOC to the renderer, TrueHD Atmos and DTS:X as APAC, in a build with the object-audio traits). No container reliably declares JOC pre-decode, so an honest `TrackInfo.isAtmos` needs a bounded decode: `AetherEngine.probeDetectingAtmos(url:/source:)` answers for a details screen without starting playback, and `LoadOptions.confirmAtmos` has the running session confirm its own tracks in the background and republish `audioTracks`. Both are opt-in and neither sits on the playback-start path. TrueHD Atmos keeps its heights too, opt-in through `LoadOptions.objectAudioRendering` in a build with the `TrueHDObjectAudio` trait ([Optional components](#optional-components)): the engine decodes the Atmos objects, renders them into a speaker bed and delivers Apple Positional Audio, which tvOS sends to an Atmos receiver as Dolby MAT and iOS plays as Spatial Audio ([details](docs/formats.md#truehd-atmos-with-heights)). DTS:X takes the same route with the `DTSXObjectAudio` trait, its four heights unfolded from the compatible 7.1 into 7.1.4 or 9.1.6 ([details](docs/formats.md#dtsx-with-heights)) |

Sources/AetherEngine/Decoder/HardwareVideoDecoder.swift

@@ -6,7 +6,7 @@ import AetherLibavformat
 import AetherLibavcodec
 import AetherLibavutil
 
-/// VTDecompressionSession-backed HEVC decoder for the SoftwarePlaybackHost pipeline.
+/// VTDecompressionSession-backed HEVC / VP9 decoder for the SoftwarePlaybackHost pipeline.
 /// Owns the decoded-frame pool, IOSurface lifetime, and session teardown explicitly
 /// (AVPlayer's opaque state grows unbounded on long 4K HDR sessions).
 /// Same surface as SoftwareVideoDecoder so the host can swap without rewiring the demux loop.
@@ -79,6 +79,7 @@ final class HardwareVideoDecoder: VideoDecodingPipeline, @unchecked Sendable {
     private var timeBase: AVRational = AVRational(num: 1, den: 90000)
     private var width: Int32 = 0
     private var height: Int32 = 0
+    private var codecName = "HEVC"
 
     /// Color metadata from codecpar, re-applied to every CVPixelBuffer.
     /// VTDecompressionSession should propagate these from SPS+hvcC but has been observed not to;
@@ -273,27 +274,25 @@ final class HardwareVideoDecoder: VideoDecodingPipeline, @unchecked Sendable {
             )
         }
 
-        guard codecpar.pointee.codec_id == AV_CODEC_ID_HEVC else {
-            throw VideoDecoderError.unsupportedCodec(id: codecpar.pointee.codec_id.rawValue)
+        let configuration = try VideoToolboxVideoFormat.make(stream: stream)
+        codecName = configuration.codecName
+        if codecpar.pointee.codec_id == AV_CODEC_ID_VP9 {
+            if #available(macOS 11.0, iOS 26.2, tvOS 26.2, visionOS 26.2, *) {
+                VTRegisterSupplementalVideoDecoderIfAvailable(kCMVideoCodecType_VP9)
+            }
+            guard VTIsHardwareDecodeSupported(kCMVideoCodecType_VP9) else {
+                throw VideoDecoderError.hardwareDecoderUnavailable(codec: "VP9")
+            }
         }
 
-        // 1. Build CMVideoFormatDescription from the hvcC extradata via
-        //    kCMFormatDescriptionExtension_SampleDescriptionExtensionAtoms (same shape AVFoundation uses for .mp4/.mkv).
-        guard let extradata = codecpar.pointee.extradata, codecpar.pointee.extradata_size > 0 else {
-            throw VideoDecoderError.noExtradata
-        }
-        let hvcCData = Data(bytes: extradata, count: Int(codecpar.pointee.extradata_size))
+        // WebM VP9 has no extradata; its vpcC record comes from the probed stream parameters.
         var fd: CMVideoFormatDescription?
-        let atomsDict: NSDictionary = ["hvcC": hvcCData]
-        let extensions: NSDictionary = [
-            kCMFormatDescriptionExtension_SampleDescriptionExtensionAtoms: atomsDict,
-        ]
         let fdStatus = CMVideoFormatDescriptionCreate(
             allocator: kCFAllocatorDefault,
-            codecType: kCMVideoCodecType_HEVC,
+            codecType: configuration.codecType,
             width: width,
             height: height,
-            extensions: extensions,
+            extensions: configuration.extensions,
             formatDescriptionOut: &fd
         )
         guard fdStatus == noErr, let formatDesc = fd else {
@@ -308,16 +307,14 @@ final class HardwareVideoDecoder: VideoDecodingPipeline, @unchecked Sendable {
         ]
 
         // 3. Pixel buffer attributes: 10-bit biplanar for HDR, 8-bit for SDR; IOSurface-backed for Metal rendering.
-        let bitsPerSample = codecpar.pointee.bits_per_raw_sample
-        let isHDRTransfer = ColorAttachments.isHDRTransfer(codecpar.pointee.color_trc)
-        let use10Bit = bitsPerSample > 8 || isHDRTransfer
+        let use10Bit = configuration.bitDepth > 8
 
         streamColor = ColorDescription(codecpar: codecpar)
         streamCodecID = codecpar.pointee.codec_id
         streamProfile = codecpar.pointee.profile
         scansHDR10Plus = codecpar.pointee.color_trc == AVCOL_TRC_SMPTE2084
         hdr10PlusFraming = .lengthPrefixed(size: NALUnitChain.lengthPrefixSize(
-            codecID: streamCodecID, extradata: extradata,
+            codecID: streamCodecID, extradata: codecpar.pointee.extradata,
             extradataSize: Int(codecpar.pointee.extradata_size)) ?? 4)
         self.colorPrimaries = ColorAttachments.primaries(codecpar.pointee.color_primaries)
         self.colorTransfer = ColorAttachments.transfer(codecpar.pointee.color_trc)
@@ -349,7 +346,7 @@ final class HardwareVideoDecoder: VideoDecodingPipeline, @unchecked Sendable {
         session = createdSession
 
         EngineLog.emit(
-            "[HardwareVideoDecoder] opened HEVC \(width)x\(height) "
+            "[HardwareVideoDecoder] opened \(codecName) \(width)x\(height) "
             + "\(use10Bit ? "10-bit" : "8-bit") "
             + "transfer=\(codecpar.pointee.color_trc.rawValue)",
             category: .swPlayback

Sources/AetherEngine/Decoder/VideoDecoderOpener.swift

@@ -0,0 +1,39 @@
+import Foundation
+import CoreMedia
+import AetherLibavformat
+
+/// Opens a preferred decoder and atomically falls back before the demux loop captures its pipeline.
+enum VideoDecoderOpener {
+    struct Result {
+        let decoder: any VideoDecodingPipeline
+        let preferredFailureDescription: String?
+
+        var usedFallback: Bool { preferredFailureDescription != nil }
+    }
+
+    static func openPreferredOrFallback(
+        preferred: any VideoDecodingPipeline,
+        makeFallback: () -> any VideoDecodingPipeline,
+        stream: UnsafeMutablePointer<AVStream>,
+        onFrame: @escaping DecodedFrameHandler
+    ) throws -> Result {
+        do {
+            try preferred.open(stream: stream, onFrame: onFrame)
+            return Result(decoder: preferred, preferredFailureDescription: nil)
+        } catch {
+            preferred.close()
+            let preferredFailureDescription = error.localizedDescription
+            let fallback = makeFallback()
+            do {
+                try fallback.open(stream: stream, onFrame: onFrame)
+                return Result(
+                    decoder: fallback,
+                    preferredFailureDescription: preferredFailureDescription
+                )
+            } catch let fallbackError {
+                fallback.close()
+                throw fallbackError
+            }
+        }
+    }
+}

Sources/AetherEngine/Decoder/VideoDecoderTypes.swift

@@ -9,8 +9,8 @@ import AetherLibavcodec
 /// (kCMSampleAttachmentKey_HDR10PlusPerFrameData format); nil for non-HDR10+ streams.
 typealias DecodedFrameHandler = @Sendable (CVPixelBuffer, CMTime, Data?) -> Void
 
-/// Common video decoder protocol. SoftwareVideoDecoder (libavcodec, AV1/VP9) and
-/// HardwareVideoDecoder (VTDecompressionSession, HEVC) both conform; the host swaps per codec without rewiring the demux loop.
+/// Common video decoder protocol. SoftwareVideoDecoder (libavcodec) and
+/// HardwareVideoDecoder (VTDecompressionSession, HEVC/VP9) both conform; the host swaps per codec without rewiring the demux loop.
 // Sendable: both conformers (SoftwareVideoDecoder, HardwareVideoDecoder) are @unchecked Sendable
 // (internally lock-guarded), so `any VideoDecodingPipeline` is safe to capture in @Sendable closures.
 protocol VideoDecodingPipeline: AnyObject, Sendable {
@@ -58,6 +58,8 @@ enum VideoDecoderError: Error, LocalizedError {
     case noCodecParameters
     case unsupportedCodec(id: UInt32)
     case noExtradata
+    case unsupportedConfiguration(codec: String, detail: String)
+    case hardwareDecoderUnavailable(codec: String)
     case formatDescriptionFailed(status: OSStatus)
     case sessionCreationFailed(status: OSStatus)
     /// `DualLayerVideoDecoder` is single use (dolby-vision-fel.md §5.8): `open` after `close` refuses
@@ -75,6 +77,10 @@ enum VideoDecoderError: Error, LocalizedError {
         case .noCodecParameters: "No codec parameters"
         case .unsupportedCodec(let id): "Unsupported video codec (id: \(id))"
         case .noExtradata: "Missing codec extradata"
+        case .unsupportedConfiguration(let codec, let detail):
+            "Unsupported \(codec) configuration: \(detail)"
+        case .hardwareDecoderUnavailable(let codec):
+            "Hardware \(codec) decoder unavailable"
         case .formatDescriptionFailed(let s): "Format description failed (\(s))"
         case .sessionCreationFailed(let s): "Decoder session failed (\(s))"
         case .decoderClosed: "Decoder is closed (single use)"