[NicolasHug]

This is part of a stack that adds demuxer-free decoding to the low-level APIs: decoding packets that don't come from a Demuxer (#1585), demuxing and decoding in different processes, and raw H.264 / HEVC streams (#1771).

---

Adds VideoStream.decoder_parameters / AudioStream.decoder_parameters, which return a VideoPacketDecoderParameters / AudioPacketDecoderParameters: everything FFmpeg knows about a stream that a decoder needs to decode its packets. VideoPacketDecoder and AudioPacketDecoder can be built directly from them, whether they come from a stream or are built by hand:

>>> Demuxer("video.mp4").streams[0].decoder_parameters
VideoPacketDecoderParameters(codec='h264', time_base=1/30000, extradata=<48 bytes>, codec_tag=828601953, profile=100, level=21, ..., width=480, height=270, pixel_format='yuv420p', ..., r_frame_rate=30000/1001, ...)

# From a demuxer (this is what stream.make_decoder() now does):
decoder = VideoPacketDecoder(demuxer.streams[0].decoder_parameters, device="cuda")

# Or by hand, for packets that don't come from a demuxer:
params = VideoPacketDecoderParameters("h264", time_base=(1, 90000), extradata=avcc)
decoder = VideoPacketDecoder(params, device="cuda", decoder="h264")

Design

• All fields public: every field of FFmpeg's AVCodecParameters is exposed, plus the stream's time_base, and for video r_frame_rate and rotation. So users building parameters by hand never need a new release to set the field their codec needs. Unset fields (None) keep FFmpeg's defaults.
• Plain values: they're frozen dataclasses of plain values, with FFmpeg's enums stored by name ("yuv420p", "bt709", "stereo"). So they pickle as is, and nothing is hidden. The one exception is a private flag for MPEG-PS packet misalignment.
• One construction path: stream.make_decoder() goes through the parameters too, so a demuxed stream and its parameters can't decode differently.
• decoder= selects a specific FFmpeg decoder, which must decode the parameters' codec.
• Validation: unknown codecs, enum names or fields raise when the decoder is built. framerate needs FFmpeg 6 and coded_side_data needs FFmpeg 6.1; setting them on older versions raises.
• CUDA needs pixel_format, width and height: NVDEC checks them before decoding to decide whether it supports the stream, and falls back to CPU decoding if it doesn't. Without them it would always fall back, silently, so leaving them out raises instead.

The class names are provisional, marked TODO_DEMUXER_FREE.

C++ changes worth a look

• DecoderParameters holds the same thing as the Python classes (an owned AVCodecParameters plus the stream-level fields). It crosses over to Python as a map of strings, with bytes in hex.
• No more Demuxer in PacketDecoder: it's built from DecoderParameters.
• Breaking for out-of-tree device interfaces: DeviceInterface::initialize_video_decoding() (and initialize_video()) take the DecoderParameters instead of an AVStream and an AVFormatContext, so out-of-tree interfaces that override it need updating. The in-tree third-party interface test doesn't override it. SingleStreamDecoder builds its DecoderParameters from its stream with the same get_decoder_parameters() as the Demuxer path.
• NVDEC bitstream filter: it used to be picked from the container name, and there may not be a container anymore. h264_mp4toannexb / hevc_mp4toannexb are now applied when the extradata isn't Annex B, using the same check as the filters themselves: Annex B extradata is empty or starts with a start code.
  • That's equivalent to the old container check for every container we handled: avcC / hvcC extradata gets the filter, and Annex B extradata didn't need it, since the filters pass Annex B through.
  • It also covers hvcC with a configurationVersion of 0, which early HEVC muxers wrote, and length-prefixed H.264 in AVI, which used to get no filter.
• MPEG-4 on NVDEC: mpeg4_unpack_bframes is now applied to every MPEG-4 stream, not just AVI ones. It passes any packet that isn't DivX-packed through unchanged.

Part of #1585.

Tests

• Values reported for H.264 (including a rotated video), HEVC and AAC streams.
• Rebuilding parameters from their public fields gives equal parameters, and decodes exactly like the originals: H.264, HEVC, AV1, VP9, MPEG-4, a rotated video, AAC, MP3, PCM. CPU and CUDA.
• Hand-built parameters work: avcC H.264 with extradata, Annex B H.264 with only codec and time_base, PCM. CPU and CUDA. The CUDA cases go through the extradata-based filter choice.
• time_base / codec_tag normalization, validation errors, decoder=, pickling, repr.

---

_{Stack created with GitHub Stacks CLI • Give Feedback 💬}

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/meta-pytorch/torchcodec/1779

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 8 New Failures, 10 Pending, 7 Unrelated Failures

As of commit 2141766 with merge base ffb098a ():

NEW FAILURES - The following jobs have failed:

• Lint / pre-commit-checks (3.12) (gh)
• Linux CPU x86 / install-and-test (3.11, 4) (gh)
  test/test_decoders.py::TestLowLevel::test_audio_decoder_parameters_round_trip_through_their_fields[asset3]
• Linux CUDA aarch64 / install-and-test (3.11, 13.0, 6) (gh)
  An unexpected error has occurred. Conda has prepared the above report.
• Linux CUDA x86 / install-and-test (3.11, 13.0, 4) (gh)
  test/test_decoders.py::TestLowLevel::test_audio_decoder_parameters_round_trip_through_their_fields[asset3]
• Linux CUDA x86 / install-and-test (3.11, 13.0, 6) (gh)
  test/test_decoders.py::TestLowLevel::test_coded_side_data_requires_ffmpeg_6_1
• Linux CUDA x86 / install-and-test (3.11, 13.2, 4) (gh)
  test/test_decoders.py::TestLowLevel::test_audio_decoder_parameters_round_trip_through_their_fields[asset3]
• Linux CUDA x86 / install-and-test (3.11, 13.2, 9) (gh)
  test/test_decoders.py::TestImageDecoder::test_cuda_jpeg_multithreaded
• Windows CPU / install-and-test (3.11, 4) (gh)
  test/smoke_test.py::TestLowLevelAPIs::test_audio_pipeline

FLAKY - The following jobs failed but were likely due to flakiness present on trunk:

• Linux CPU x86 / install-and-test (3.11, 6) (gh) (detected as infra flaky with no log or failing log classifier)
• Linux CPU x86 / install-and-test-no-ffmpeg (3.11) (gh) (detected as infra flaky with no log or failing log classifier)
• Linux CUDA aarch64 / install-and-test (3.11, 13.0, 4) (gh) (detected as infra flaky with no log or failing log classifier)
• Linux CUDA x86 / install-and-test (3.11, 13.2, 6) (gh) (detected as infra flaky with no log or failing log classifier)
• Linux CUDA x86 / install-and-test-no-ffmpeg (gh) (detected as infra flaky with no log or failing log classifier)
• MacOS / install-and-test (3.11, 4) (gh) (detected as infra flaky with no log or failing log classifier)
• Windows CUDA / install-and-test-on-cpu-only-machine (gh) (detected as infra flaky with no log or failing log classifier)

This comment was automatically generated by Dr. CI and updates every 15 minutes.