[NicolasHug]

This is part of a stack that adds demuxer-free decoding to the low-level APIs: decoding packets that don't come from a Demuxer (#1585), demuxing and decoding in different processes, and raw H.264 / HEVC streams (#1771).

---

VideoPacketDecoder and AudioPacketDecoder can now be built directly from the decoder parameters added in #1779, whether they come from a stream or are built by hand:

# From a demuxer (this is what stream.make_decoder() now does):
decoder = VideoPacketDecoder(demuxer.streams[0].decoder_parameters, device="cuda")

# Or by hand, for packets that don't come from a demuxer:
params = VideoPacketDecoderParameters("h264", time_base=(1, 90000), extradata=avcc)
decoder = VideoPacketDecoder(params, device="cuda", decoder="h264")

• One construction path: stream.make_decoder() goes through the same path, so a demuxed stream and its parameters can't decode differently.
• decoder= selects a specific FFmpeg decoder, which must decode the parameters' codec.
• Validation: unknown codecs, enum names or fields raise when the decoder is built. framerate needs FFmpeg 6 and coded_side_data needs FFmpeg 6.1; setting them on older versions raises.

C++ changes worth a look

• No more Demuxer in PacketDecoder: it's built from DecoderParameters.
• Breaking for out-of-tree device interfaces: DeviceInterface::initialize_video_decoding() (and initialize_video()) take the DecoderParameters instead of an AVStream and an AVFormatContext, so out-of-tree interfaces that override it need updating. The in-tree third-party interface test doesn't override it. SingleStreamDecoder builds its DecoderParameters from its stream with the same get_decoder_parameters() as the Demuxer path.
• NVDEC bitstream filter: it used to be picked from the container name, and there may not be a container anymore. h264_mp4toannexb / hevc_mp4toannexb are now applied when the extradata isn't Annex B, using the same check as the filters themselves: Annex B extradata is empty or starts with a start code.
  • That's equivalent to the old container check for every container we handled: avcC / hvcC extradata gets the filter, and Annex B extradata didn't need it, since the filters pass Annex B through.
  • It also covers hvcC with a configurationVersion of 0, which early HEVC muxers wrote, and length-prefixed H.264 in AVI, which used to get no filter.
• MPEG-4 on NVDEC: mpeg4_unpack_bframes is now applied to every MPEG-4 stream, not just AVI ones. It passes any packet that isn't DivX-packed through unchanged.

Part of #1585.

Tests

• Parameters rebuilt from their public fields decode exactly like the originals: H.264, HEVC, AV1, VP9, MPEG-4, a rotated video, AAC, MP3, PCM. CPU and CUDA.
• Hand-built parameters work: avcC H.264 with extradata, Annex B H.264 with only codec and time_base, PCM. CPU and CUDA. The CUDA cases go through the extradata-based filter choice.
• Validation errors and decoder=.

---

_{Stack created with GitHub Stacks CLI • Give Feedback 💬}

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/meta-pytorch/torchcodec/1780

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 11 New Failures, 8 Unrelated Failures

As of commit 2141766 with merge base ffb098a ():

NEW FAILURES - The following jobs have failed:

• Lint / pre-commit-checks (3.12) (gh)
• Linux CPU x86 / install-and-test (3.11, 4) (gh)
  test/test_decoders.py::TestLowLevel::test_audio_decoder_parameters_round_trip_through_their_fields[asset3]
• Linux CPU x86 / install-and-test (3.11, 6) (gh)
  test/test_decoders.py::TestLowLevel::test_coded_side_data_requires_ffmpeg_6_1
• Linux CUDA aarch64 / install-and-test (3.11, 13.0, 4) (gh)
  test/smoke_test.py::TestLowLevelAPIs::test_audio_pipeline
• Linux CUDA x86 / install-and-test (3.11, 13.0, 4) (gh)
  test/test_decoders.py::TestLowLevel::test_audio_decoder_parameters_round_trip_through_their_fields[asset3]
• Linux CUDA x86 / install-and-test (3.11, 13.0, 6) (gh)
  test/test_decoders.py::TestLowLevel::test_coded_side_data_requires_ffmpeg_6_1
• Linux CUDA x86 / install-and-test (3.11, 13.2, 4) (gh)
  test/test_decoders.py::TestLowLevel::test_audio_decoder_parameters_round_trip_through_their_fields[asset3]
• Linux CUDA x86 / install-and-test (3.11, 13.2, 6) (gh)
  test/test_decoders.py::TestLowLevel::test_coded_side_data_requires_ffmpeg_6_1
• MacOS / install-and-test (3.11, 4) (gh)
  test/smoke_test.py::TestLowLevelAPIs::test_audio_pipeline
• Windows CPU / install-and-test (3.11, 4) (gh)
  test/smoke_test.py::TestLowLevelAPIs::test_audio_pipeline
• Windows CUDA / install-and-test (3.11, 13.2, 4) (gh)
  test/smoke_test.py::TestLowLevelAPIs::test_audio_pipeline

FLAKY - The following jobs failed but were likely due to flakiness present on trunk:

• Linux CPU x86 / install-and-test-no-ffmpeg (3.11) (gh) (detected as infra flaky with no log or failing log classifier)
• Linux CUDA x86 / install-and-test-no-ffmpeg (gh) (detected as infra flaky with no log or failing log classifier)
• Windows CUDA / install-and-test (3.11, 13.0, 7) (gh) (detected as infra flaky with no log or failing log classifier)

BROKEN TRUNK - The following jobs failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

• Windows CUDA / install-and-test (3.11, 13.0, 4) (gh) (trunk failure)
• Windows CUDA / install-and-test (3.11, 13.0, 6) (gh) (trunk failure)
  Unable to download artifact(s): Artifact not found
• Windows CUDA / install-and-test (3.11, 13.0, 8) (gh) (trunk failure)
• Windows CUDA / install-and-test (3.11, 13.0, 9) (gh) (trunk failure)
• Windows CUDA / install-and-test-on-cpu-only-machine (gh) (trunk failure)
  Unable to download artifact(s): Artifact not found

This comment was automatically generated by Dr. CI and updates every 15 minutes.