Skip to content

feat(video): decode supported VP9 streams with VideoToolbox - #878

Open
jonluca wants to merge 1 commit into
superuser404notfound:mainfrom
jonluca:feat/vp9-videotoolbox-decoding
Open

jonluca wants to merge 1 commit into
superuser404notfound:mainfrom
jonluca:feat/vp9-videotoolbox-decoding

Conversation

@jonluca

@jonluca jonluca commented Oct 25, 2026

Copy link
Copy Markdown
Contributor

Summary

Use VideoToolbox hardware decoding for supported VP9 streams while keeping the software playback host's presentation path. VP9 previously always used libavcodec even on hardware that can decode it directly; AVPlayer's HLS route remains unsuitable for VP9.

What changed

  • Synthesize WebM's missing vpcC record from FFmpeg's probed profile, depth, chroma, range and color parameters.
  • Register the supplemental VP9 decoder when available and require an actual hardware session for 8-bit profile 0 / 10-bit profile 2, both 4:2:0.
  • Close a failed hardware candidate and open libavcodec before the read loop captures the pipeline; also close and propagate a failed fallback.
  • Preserve upstream HEVC probing, Dolby Vision selection, HDR10+ packet side data and frame-progress behavior. Existing decoder diagnostics now naturally report the selected VP9 implementation.
  • Add configuration/selection/fallback regressions and exercise decoder identity plus decoded frames through a real VP9 load and reload. Update format and architecture documentation.

Test plan

  • Device / OS: Apple M4 Max, macOS 27.0.1, Xcode 27.1.
  • Media: synthetic WebM, VP9 profile 0, 1920×1080 at 24 fps, 5 seconds, no audio, SDR. Generated with the same command used by Scripts/fetch-fixtures.sh for vp9.webm.
  • 73 tests across 9 suites pass (three unrelated real-file cases skip because their fixtures are absent). Coverage includes vpcC layout, ten-bit HDR/full-range configuration, unsupported profiles/depth/chroma/siting, decoder choice, cleanup on failed preferred/fallback opens, and real VP9 hardware load/reload with decoded frames.
  • aetherctl play --seconds 4 Fixtures/vp9.webm passes: VideoToolbox VP9 opens, frames are decoded, and the clock remains at 1.00x. This is an output/clock smoke check without a bound onscreen surface.
  • AetherEngine builds with -warnings-as-errors.
  • No iOS/tvOS playback or hardware performance claim is made.
  • python3 Scripts/check-doc-links.py and git diff --check pass.

Checklist

  • CHANGELOG.md updated
  • Conventional commit
  • Feature lives in the engine
  • No public API change

Copilot AI balanced review requested due to automatic review settings October 25, 2026 16:30

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Hardware-dependent decoding and presentation need human validation beyond the reported single-device, 8-bit macOS case, particularly for 10-bit VP9 and other Apple platforms.

No open issues 🎉

What changed in this PR

Adds VideoToolbox hardware decoding for supported VP9 streams within the existing software playback host, retaining libavcodec fallback and the current presentation path.

Changes:

  • Synthesizes VP9 configuration records and supports hardware sessions for 8-bit profile 0 and 10-bit profile 2, both 4:2:0.
  • Opens software fallback and cleans up failed decoders before packet reading begins.
  • Adds configuration, fallback, and load/reload regressions; updates playback documentation.
File Description
Tests/​AetherEngineTests/​SoftwareDecoderSelectionTests.swift Tests VP9 configuration, selection, and fallback cleanup.
Tests/​AetherEngineTests/​ActiveVideoDecoderTests.swift Checks VP9 decoder identity and frames across load/reload.
Sources/​AetherEngine/​Native/​SoftwarePlaybackHost.swift Integrates VP9 hardware selection and software fallback.
Sources/​AetherEngine/​Decoder/​VideoToolboxVideoFormat.swift Builds codec descriptions and validates VP9 configurations.
Sources/​AetherEngine/​Decoder/​VideoDecoderTypes.swift Adds configuration and hardware-availability errors.
Sources/​AetherEngine/​Decoder/​VideoDecoderOpener.swift Handles preferred decoder opening and fallback cleanup.
Sources/​AetherEngine/​Decoder/​HardwareVideoDecoder.swift Enables VP9 registration and hardware session setup.
README.md Updates the VP9 support summary.
docs/​formats.md Documents supported VP9 formats and fallback behavior.
docs/​architecture.md Explains VP9 integration with the software host.
CHANGELOG.md Records VP9 hardware decoding support.

Review effort: Balanced

@jonluca
jonluca force-pushed the feat/vp9-videotoolbox-decoding branch from 66b3b62 to df8fc1f Compare October 25, 2026 18:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants