[NicolasHug]

This is part of a stack that adds demuxer-free decoding to the low-level APIs: decoding packets that don't come from a Demuxer (#1585), demuxing and decoding in different processes, and raw H.264 / HEVC streams (#1771).

---

Adds Packet.from_bytes() and Packet.from_tensor() to build packets from compressed data that doesn't come from a Demuxer, e.g. packets from the network or from a database, as in #1585:

params = VideoPacketDecoderParameters("h264", time_base=(1, 1_000_000_000))
decoder = VideoPacketDecoder(params, device="cuda")

for data, pts in rows:  # from a keyframe, in decoding order
    for raw_frame in decoder.decode(Packet.from_bytes(data, pts=pts)):
        if raw_frame.pts == target_pts:
            ...

• Copying: the data is copied. FFmpeg needs zeroed padding past the end of packet data, which user buffers don't have. Zero-copy could come later, through new entry points into memory we allocate.
• Optional fields: pts, dts, duration, is_key_frame and side_data are all optional. Decoders only need the data, plus pts to report frame timestamps. In particular dts isn't needed for streams with B-frames: there's a test for that, CPU and CUDA.
• Accessors: packets also expose pts, dts, duration, is_key_frame, data and side_data.
• Integer pts on frames: RawFrame.pts and RawAudioSamples.pts now expose the decoded frames' pts as integers, in the new VideoPacketDecoder.time_base / AudioPacketDecoder.time_base, or None when unknown. Packets go in with integer timestamps, so this lets users match a decoded frame to the timestamp they asked for exactly, rather than reproducing our float conversion bit for bit.

Addresses the use case of #1585.

Tests

• Packets rebuilt from demuxed packets' bytes decode identically, CPU and CUDA.
• Packets with only pts, or no timestamps at all, decode fine.
• A Add Packet.from_tensor to build packets from raw data #1585-style test: decode targets from keyframe-to-target packet windows, reusing one decoder.
• Accessors, copy semantics, validation errors, and integer pts matching the scanned index.

---

_{Stack created with GitHub Stacks CLI • Give Feedback 💬}

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/meta-pytorch/torchcodec/1782

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 18 New Failures, 8 Unrelated Failures

As of commit 86e1dcf with merge base ffb098a ():

NEW FAILURES - The following jobs have failed:

• Lint / pre-commit-checks (3.12) (gh)
• Linux CPU x86 / install-and-test (3.11, 4) (gh)
  test/test_decoders.py::TestLowLevel::test_audio_decoder_parameters_round_trip_through_their_fields[asset3]
• Linux CPU x86 / install-and-test (3.11, 6) (gh)
  test/test_decoders.py::TestLowLevel::test_coded_side_data_requires_ffmpeg_6_1
• Linux CUDA aarch64 / install-and-test (3.11, 13.0, 4) (gh)
  test/smoke_test.py::TestLowLevelAPIs::test_audio_pipeline
• Linux CUDA aarch64 / install-and-test (3.11, 13.0, 9) (gh)
  test/test_decoders.py::TestLowLevel::test_decode_in_another_process
• Linux CUDA x86 / install-and-test (3.11, 13.0, 4) (gh)
  test/test_decoders.py::TestLowLevel::test_decode_in_another_process
• Linux CUDA x86 / install-and-test (3.11, 13.0, 6) (gh)
  test/test_decoders.py::TestLowLevel::test_decode_in_another_process
• Linux CUDA x86 / install-and-test (3.11, 13.0, 7) (gh)
  test/test_decoders.py::TestLowLevel::test_decode_in_another_process
• Linux CUDA x86 / install-and-test (3.11, 13.0, 8) (gh)
  test/test_decoders.py::TestLowLevel::test_decode_in_another_process
• Linux CUDA x86 / install-and-test (3.11, 13.0, 9) (gh)
  test/test_decoders.py::TestLowLevel::test_decode_in_another_process
• Linux CUDA x86 / install-and-test (3.11, 13.2, 4) (gh)
  test/test_decoders.py::TestLowLevel::test_decode_in_another_process
• Linux CUDA x86 / install-and-test (3.11, 13.2, 6) (gh)
  test/test_decoders.py::TestLowLevel::test_decode_in_another_process
• Linux CUDA x86 / install-and-test (3.11, 13.2, 7) (gh)
  test/test_decoders.py::TestLowLevel::test_decode_in_another_process
• Linux CUDA x86 / install-and-test (3.11, 13.2, 8) (gh)
  test/test_decoders.py::TestLowLevel::test_decode_in_another_process
• Linux CUDA x86 / install-and-test (3.11, 13.2, 9) (gh)
  test/test_decoders.py::TestLowLevel::test_decode_in_another_process
• MacOS / install-and-test (3.11, 4) (gh)
  test/smoke_test.py::TestLowLevelAPIs::test_audio_pipeline
• Windows CPU / install-and-test (3.11, 4) (gh)
  test/smoke_test.py::TestLowLevelAPIs::test_audio_pipeline
• Windows CUDA / install-and-test (3.11, 13.2, 4) (gh)
  test/smoke_test.py::TestLowLevelAPIs::test_audio_pipeline

FLAKY - The following jobs failed but were likely due to flakiness present on trunk:

• Linux CPU x86 / install-and-test-no-ffmpeg (3.11) (gh) (detected as infra flaky with no log or failing log classifier)
• Linux CUDA x86 / install-and-test-no-ffmpeg (gh) (detected as infra flaky with no log or failing log classifier)
• Windows CUDA / install-and-test (3.11, 13.0, 4) (gh) (similar failure)
  Unable to download artifact(s): Artifact not found
• Windows CUDA / install-and-test (3.11, 13.0, 8) (gh) (similar failure)
  Unable to download artifact(s): Artifact not found
• Windows CUDA / install-and-test (3.11, 13.0, 9) (gh) (similar failure)
  Unable to download artifact(s): Artifact not found

BROKEN TRUNK - The following jobs failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

• Windows CUDA / install-and-test (3.11, 13.0, 6) (gh) (trunk failure)
  Unable to download artifact(s): Artifact not found
• Windows CUDA / install-and-test (3.11, 13.0, 7) (gh) (trunk failure)
  Unable to download artifact(s): Artifact not found
• Windows CUDA / install-and-test-on-cpu-only-machine (gh) (trunk failure)
  Unable to download artifact(s): Artifact not found

This comment was automatically generated by Dr. CI and updates every 15 minutes.