ffmpeg: ffmpeg_dec: core module framework and audio decoders (FLAC, MP3, AAC, Opus) by lgirdwood · Pull Request #11308 · thesofproject/sof

lgirdwood · 2026-10-14T14:14:35Z

Overview

This is PR 1 in the series splitting monolithic PR #10990 into modular, reviewable PRs rebased on main.

This PR introduces the Core Engine & Audio Decoders (ffmpeg_dec) module, enabling playback of compressed audio elementary streams (FLAC, MP3, AAC-LC, and Opus) on SOF hardware platforms (including PTL, LNL, MTL, NVL, TGL, and WCL).

The module sources libavcodec and libavutil directly from the official SOF fork thesofproject/FFmpeg via the West manifest (west.yml).

Subsequent PRs in the series will build on this core:

PR 2: WebRTC audio processing modules (AEC, NS, VAD, RNNoise)
PR 3: FFmpeg audio encoders (ffmpeg_enc: MP3, AAC)
PR 4: Full topology integration for compressed playback & capture
PR 5: Documentation & tests
PR 6: FFmpeg audio filters (afftdn, alimiter)

Changes in this PR

1. West Manifest Integration (`west.yml`)

Adds ffmpeg pointing to official fork thesofproject/FFmpeg pinned to the sof-hifi branch revision (08ab4baa3400d55be75be1280f6119e210ce354e).
Placed under modules/audio/ffmpeg.

2. Audio Processing Module (`src/audio/ffmpeg_dec`)

Core module framework: implements SOF processing module interface supporting both built-in (=y) and LLEXT (=m) modular builds, plus a stub backend (CONFIG_COMP_FFMPEG_DEC_STUB) for CI validation without external dependencies.
Audio Decoders:
- FLAC: lossless integer decoder (fixed-point arithmetic, minimal footprint).
- MP3: MPEG-1/2 Layer III decoder via mpegaudio parser and float/fixed synthesis.
- AAC-LC: Low Complexity AAC decoder.
- Opus: SILK+CELT decoder.
Private libc and newlib double math:
- ffmpeg_dec-builtin-libc.c: provides isolated libc symbols required by FFmpeg without conflicting with Zephyr minimal libc.
- Vendored newlib math archive (libnewlib_m.a) compiled with -mlongcalls -mtext-section-literals to satisfy double-precision math without calling into unsupported host libc.
DSP Heap Allocator:
- ffmpeg_dec-builtin-alloc.c: wraps malloc/free/realloc/calloc onto the unshared DSP heap (.heap_mem), allowing large decoder working sets (e.g. AAC-LC MDCT tables) without overflowing the common-libc arena.
Xtensa Compiler & Longcalls Compliance:
- Passes -mlongcalls -mtext-section-literals to prevent call8 out-of-range relocation errors across segmented DSP memory.
- Soft-float scalar math options for platforms with scalar FPU unconfigured.
IPC & Base Firmware Capabilities:
- Adds SND_AUDIOCODEC_OPUS_RAW (0x11) definition to src/include/ipc/compress_params.h.
- Advertises FLAC, MP3, AAC, and Opus capabilities in src/audio/base_fw.c when configured.
- Rimage configuration registration across PTL, LNL, MTL, NVL, TGL, TGL-H, and WCL.

3. Pipeline & Topology Definitions

include/pipelines/cavs/compr-playback-ffmpeg.conf: compress playback pipeline definition routing through ffmpeg_dec.
platform/intel/compr-ffmpeg.conf: platform instance with FLAC STREAMINFO setup control, gated on COMPR_FFMPEG.
Development topology build targets in development/tplg-targets.cmake:
- sof-hda-generic-ffmpeg-compr (FLAC)
- sof-hda-generic-ffmpeg-mp3-compr (MP3)
- sof-hda-generic-ffmpeg-aac-compr (AAC-LC)

Verification & Build Results

Baseline PTL Build:
- xtensa-build-zephyr.py -z -p ptl: PASS (code 0)
Stub Backend Build:
- xtensa-build-zephyr.py -z -p -o ffmpeg_dec_stub.conf ptl: PASS (code 0)
Real Decoders (PTL / ACE 3.0):
- ffmpeg_dec_flac_y.conf: PASS (code 0)
- ffmpeg_dec_mp3_y.conf: PASS (code 0)
- ffmpeg_dec_aac_y.conf: PASS (code 0)
- ffmpeg_dec_opus_y.conf: PASS (code 0)
TGL (CAVS 2.5):
- xtensa-build-zephyr.py -z -p -o ffmpeg_dec_stub.conf tgl: PASS (code 0)
Topology Compilation:
- Built targets: topology2_dev_sof-hda-generic-ffmpeg-compr, topology2_dev_sof-hda-generic-ffmpeg-mp3-compr, topology2_dev_sof-hda-generic-ffmpeg-aac-compr: PASS (code 0)

Copilot

🟡 Changes recommended

Codec selection, memory sizing, allocator concurrency, stream draining, metadata handling, and modular math linkage contain correctness and reliability defects.

15 open findings

Make allocator binding instance-safe · New
Prevent unsynchronized image-wide malloc interception · New
Link real libm for modular FFmpeg decoder · New
Zero-initialize attribute buffer before dispatch · New
Increase modular heap for AAC decoder working set · New
Advertise only codecs enabled by Kconfig · New
Recreate parser during decoder reset · New
Support maximum FLAC and Opus frame sizes · New
Parse per-instance codec ID from IPC4 setup payload · New
Incrementally discard oversized ID3v2 tags · New
Parse oversized FLAC metadata incrementally · New
Flush parser and decoder before signaling EOS · New
Clamp final playback copy to drain boundary · New
Avoid hard-coded FLAC STREAMINFO in topology · New
Remove ineffective extra EDF stack reservation · New

What changed in this PR

Adds SOF’s FFmpeg-backed compressed-audio decoder framework for FLAC, MP3, AAC-LC, and Opus, including firmware, IPC, topology, build, and platform integration.

Changes:

Adds built-in/LLEXT decoder implementations, allocators, conversion, and private math/libc support.
Adds compressed-playback topology targets and codec capability advertisement.
Extends IPC drain handling, cross-core operations, manifests, and platform module packaging.

File	Description
`west.yml`	Adds the pinned SOF FFmpeg fork.
`uuid-registry.txt`	Registers the decoder UUID.
`zephyr/Kconfig`	Adjusts EDF stack configuration.
`zephyr/include/rtos/idc.h`	Adds no-timeout IDC mode and payload alignment.
`src/audio/CMakeLists.txt`	Includes the decoder build.
`src/audio/Kconfig`	Exposes decoder configuration.
`src/audio/base_fw.c`	Advertises compressed codec capabilities.
`src/audio/host-zephyr.c`	Adds compressed-stream drain tracking.
`src/audio/ffmpeg_dec/*`	Implements decoder framework, backends, allocators, conversion, build integration, and metadata.
`src/audio/ffmpeg_dec/libm/*`	Vendors private newlib math sources.
`src/idc/idc.c`	Handles cross-core attributes and reset operations.
`src/idc/zephyr_idc.c`	Implements indefinite blocking IDC waits.
`src/include/ipc/compress_params.h`	Adds the raw Opus codec identifier.
`tools/topology/topology2/sof-hda-generic.conf`	Includes FFmpeg compressed playback.
`tools/topology/topology2/cavs-mixin-mixout-hda.conf`	Adds FFmpeg codec selection and inclusion.
`tools/topology/topology2/development/tplg-targets.cmake`	Adds FLAC, MP3, and AAC development targets.
`tools/topology/topology2/include/components/ffmpeg_dec*`	Defines decoder widgets and FLAC setup data.
`tools/topology/topology2/include/pipelines/cavs/compr-playback-ffmpeg.conf`	Defines the compressed playback pipeline.
`tools/topology/topology2/platform/intel/compr-default.conf`	Adds decoder memory defaults.
`tools/topology/topology2/platform/intel/compr-ffmpeg.conf`	Instantiates the FFmpeg compressed pipeline.
`tools/rimage/config/{lnl,mtl,nvl,ptl,tgl,tgl-h,wcl}.toml.h`	Registers the module across Intel platforms.

🧠 Review effort: Balanced

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

+static struct processing_module *ffmpeg_dec_alloc_mod;
+
+void ffmpeg_dec_libc_bind(struct processing_module *mod)
+{
+	ffmpeg_dec_alloc_mod = mod;


+	/*
+	 * Lazy-init on first use so that EVERY malloc in the image (including
+	 * pre-bind boot allocations) is served from our heap. This makes the
+	 * common-libc arena dead weight, so it can be shrunk to a token size
+	 * and the reclaimed ucram handed to this heap instead.
+	 */
+	ffmpeg_dec_heap_ensure();
+
+	{
+		void *p = sys_heap_aligned_alloc(&ffmpeg_dec_heap,
+						 FFMPEG_DEC_ALLOC_ALIGN, size);


+
+	# avfilter graph backend (PCM effects, e.g. afftdn noise reduction). Pulls
+	# libavfilter into the link; it must precede the libs it depends on.
+	set(ffmpeg_dec_libs avcodec swresample avutil)


@@ -121,6 +122,8 @@ static int idc_get_attribute(uint32_t comp_id)
 	struct ipc_comp_dev *ipc_dev;
 	struct idc_payload *idc_payload;
 	struct get_attribute_remote_payload *get_attr_payload;
+	struct ipc4_base_module_cfg local_cfg;


+	# instead of COMPRESSED_1 (they share the COMPR_* pcm/pipeline defines).
+	COMPR_FFMPEG		false
+	COMPR_FFMPEG_DEC_STACK	65536
+	COMPR_FFMPEG_DEC_HEAP	49152


+		if (off + 4 > req) {
+			source_release_data(src, 0);
+			comp_err(dev, "FLAC header exceeds %zu-byte input block", req);
+			return -EINVAL;
+		}
+		hdr = cd->in_buf[off];
+		len = ((uint32_t)cd->in_buf[off + 1] << 16) |
+		      ((uint32_t)cd->in_buf[off + 2] << 8) |
+		      (uint32_t)cd->in_buf[off + 3];
+		off += 4 + len;
+		if (hdr & 0x80)		/* last-metadata-block flag */
+			break;
+	}
+
+	if (off > req) {
+		source_release_data(src, 0);
+		comp_err(dev, "FLAC header (%zu) exceeds input block (%zu)", off, req);
+		return -EINVAL;


+	avail = source_get_data_available(src);
+	if (!avail) {
+#if CONFIG_IPC_MAJOR_4
+		/*
+		 * Under a compress drain the host stops delivering data once the
+		 * file has been fully consumed. With no input left and no staged
+		 * PCM, the stream is flushed - complete the drain.
+		 */
+		if (host_eos)
+			ffmpeg_dec_signal_eos(mod, sink);


+	/*
+	 * Compress drain over a free-running HDA gateway: the write pointer never
+	 * settles so avail_samples cannot reach 0. The host instead tells us the
+	 * committed byte total (drain_bytes); once we have delivered that many
+	 * bytes downstream the real input is exhausted - flag end-of-stream.
+	 */
+	if (hd->drain_bytes && hd->total_data_processed >= hd->drain_bytes) {


+# This particular STREAMINFO describes the bring-up test clip: 48000 Hz, 2 ch,
+# 16-bit, blocksize 4096 (generated with:
+#   sox -n -r 48000 -c 2 -b 16 flactest.flac synth 3 sine 440
+# then bytes 8..41 of the .flac file). For a different clip, regenerate the
+# 34 STREAMINFO bytes below and update the size field is NOT needed (34 fixed).


+	default 32768 if FFMPEG_DEC_FLOAT_MATH
 	default 8192
 	help
 	  EDF scheduler work-queue thread stack size. Keep a power of 2.

+	  SOF runs module init and prepare (including the backend configure
+	  step) on this EDF work-queue thread. The ffmpeg_dec lossy decoders
+	  (MP3/AAC/Opus) build large cos/pow tables in avcodec_open2(), whose
+	  call chain is ~13 KiB deep and overflows the 8 KiB default -- spilling
+	  into the adjacent thread's stack and faulting on its next context
+	  switch. FFMPEG_DEC_FLOAT_MATH is auto-selected by exactly those
+	  decoders, so bump the default to 32 KiB for them; the integer FLAC
+	  decoder does not need it and keeps 8 KiB.


intel-sofci · 2026-10-14T18:29:34Z

PR 11308: test results

Run date: 2026-10-14 17:57 UTC

Tested commit: 618c239f314f71e8b94c03293544f77362962b49

struct idc_payload lives in coherent (SHARED_DATA) memory and is shared across cores, but it is only a byte array with no explicit alignment, so the linker was free to place it (and every per-core slot) on an odd address. Handlers that read the payload as a wider type -- e.g. the cross-core get_attribute path, which reads a uint32 type field -- then take an EXCCAUSE 9 (load/store alignment) fault when a component runs on a secondary core (ptl/ace30). Align the payload to the cache line. That is the granularity the payload is already built around (IDC_MAX_PAYLOAD_SIZE == DCACHE_LINE_SIZE * 2) and it guarantees every per-core slot starts word-aligned. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

The remote (cross-core) get_attribute handler passed the caller coherent (uncached) IDC buffer straight to comp_get_attribute(), which memcpy()s a whole ipc4_base_module_cfg into it. On Xtensa the compiler lowers that struct copy to wide accesses (AE_L32X2 / 64-bit) that the uncached window does not support, faulting EXCCAUSE 9 -- so binding a component that lives on a secondary core (e.g. ffmpeg_dec offloaded to core 1 on ptl/ace30) crashed. Have the handler fill a local cacheable copy and publish it to the coherent buffer with aligned 32-bit word stores (ipc4_coherent_copy32); do the same for the read-back on the initiator side. Only COMP_ATTR_BASE_CONFIG is supported remotely, so the fixed-size local copy is always correct. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

A remote component prepare() can be expensive: a DP decoder one-time codec open builds large trig tables -- measured ~580 ms for mp3 on a secondary core. Waiting for it stalls the initiator core IPC/EDF thread for that whole time, and with the default CONFIG_IDC_TIMEOUT_US the blocking k_p4wq_wait() instead times out mid-prepare (-EAGAIN) and races pipeline teardown into a crash. Send the prepare IDC non-blocking. Ordering is preserved without the wait: each core drains its IDC work through a single in-order p4wq worker, so the prepare completes before the cross-core trigger (START) queued behind it. For a compress stream the START IPC also arrives only after the app first write, giving the remote prepare time to finish off the critical path. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

comp_reset_remote() sent the cross-core IDC_MSG_RESET with IDC_BLOCKING, which only waits CONFIG_IDC_TIMEOUT_US (15 ms) before returning -EAGAIN. A data-processing (DP) module reset runs schedule_task_cancel() -> k_thread_join() on the target core and cannot return until the DP thread finishes its in-flight process() call. That call is bounded but can be long during a codec cold-start (a deferred avcodec_open2 table build, up to ~1.4 s for aac, or the first decode after it), far exceeding the IDC timeout. When it timed out the component was left un-reset (not COMP_STATE_READY), so pipeline_reset() failed and the host-side free was then refused with an invalid-state error and the remote core could not be disabled - every teardown wedged until reboot. The caller (pipeline_reset -> host) depends on the reset actually being finished before it frees the component and powers down the remote core, so the reset must stay synchronous. Add IDC_BLOCKING_NO_TIMEOUT, which is IDC_BLOCKING but waits K_FOREVER for the handler to complete, and use it for comp_reset_remote(). A normal end-of-stream teardown joins an idle DP thread and returns immediately, so this only waits when it must. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…count The HDA compress gateway free-runs its write pointer circularly and re-presents already-played data, so host_handle_eos() never sees dma_stat.pending_length reach 0 and the source buffer is never flagged END_OF_STREAM. A compress drain therefore never completes. On DRAIN the kernel now hands the host copier the total committed gateway byte count via a new SET_LARGE_CONFIG param (IPC4_COPIER_MODULE_CFG_PARAM_DRAIN_BYTES = 100), stored in host_data.drain_bytes. Once the copier has delivered that many bytes (total_data_processed >= drain_bytes), host_handle_eos() sets the source buffer to END_OF_STREAM and stops the copy, giving the pipeline a correct end of input. drain_bytes is cleared on reset. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Add the FFmpeg project to west.yml using the official thesofproject fork repository (thesofproject/FFmpeg) pinned to the sof-hifi branch revision (08ab4baa3400d55be75be1280f6119e210ce354e). This supplies libavcodec and libavutil sources for the ffmpeg_dec audio decoder module. Signed-off-by: Liam Girdwood <liam.r.girdwood@linux.intel.com>

Define SND_AUDIOCODEC_OPUS_RAW (0x11) and update SND_AUDIOCODEC_MAX so that compressed Opus playback streams can be advertised and instantiated across SOF IPC interfaces. Signed-off-by: Liam Girdwood <liam.r.girdwood@linux.intel.com>

Advertise supported compressed audio playback formats (FLAC, MP3, AAC-LC, and Opus) to the host in get_codec_info() when their respective CONFIG_FFMPEG_DEC_* symbols are enabled. Signed-off-by: Liam Girdwood <liam.r.girdwood@linux.intel.com>

… AAC, Opus) Add the ffmpeg_dec audio processing module providing compressed audio decoding for SOF pipelines using FFmpeg libavcodec. Key capabilities: - Module framework supporting both built-in (=y) and LLEXT (=m) loadable module builds, as well as a dependency-free stub backend for CI. - Support for FLAC (lossless fixed-point), MP3 (MPEG-1/2 Layer III), AAC-LC (Low Complexity), and Opus audio elementary streams. - Built-in private libc and newlib double-math archive compiled with -mlongcalls to satisfy libavcodec mathematical functions without polluting the minimal libc namespace. - DSP heap wrap allocator (ffmpeg_dec-builtin-alloc.c) to handle large decoder working sets without exceeding common-libc arena limits. - End-of-stream (EOS) signaling and compress drain handling. - Rimage module manifest toml and configuration registration for Intel platforms (PTL, LNL, MTL, NVL, TGL, TGL-H, WCL). Signed-off-by: Liam Girdwood <liam.r.girdwood@linux.intel.com>

Add topology definitions for driving compressed audio playback through the ffmpeg_dec widget instead of the legacy Cadence decoder: - New pipeline: include/pipelines/cavs/compr-playback-ffmpeg.conf - Platform instances: platform/intel/compr-ffmpeg.conf gated on COMPR_FFMPEG - Decoder component definition: include/components/ffmpeg_dec.conf - Configurable memory budgets (COMPR_FFMPEG_DEC_STACK / HEAP) in compr-default.conf - Added development topology targets: - sof-hda-generic-ffmpeg-compr (FLAC) - sof-hda-generic-ffmpeg-mp3-compr (MP3) - sof-hda-generic-ffmpeg-aac-compr (AAC-LC) Signed-off-by: Liam Girdwood <liam.r.girdwood@linux.intel.com>

lgirdwood requested review from abonislawski, dbaluta, iuliana-prodan, jsarha, kv2019i, lbetlej, lyakh, mmaka1, pblaszko, plbossart, ranj063 and tmleman as code owners October 14, 2026 14:14

Copilot AI balanced review requested due to automatic review settings October 14, 2026 14:14

lgirdwood mentioned this pull request Oct 14, 2026

WIP: Audio Processing & Codec Suite: FFmpeg dec/enc, WebRTC APM, and DSP offload filters #10990

Draft

Copilot started reviewing on behalf of lgirdwood October 14, 2026 14:15 View session

Copilot AI reviewed Oct 14, 2026

View reviewed changes

lrgirdwo and others added 10 commits October 14, 2026 20:28

lgirdwood force-pushed the topic/ffmpeg-decoders branch from 618c239 to 76ef421 Compare October 14, 2026 19:32

Navigation Menu

Repository navigation

ffmpeg: ffmpeg_dec: core module framework and audio decoders (FLAC, MP3, AAC, Opus) - #11308

ffmpeg: ffmpeg_dec: core module framework and audio decoders (FLAC, MP3, AAC, Opus)#11308
lgirdwood wants to merge 10 commits into
thesofproject:mainfrom
lgirdwood:topic/ffmpeg-decoders

lgirdwood commented Oct 14, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

intel-sofci commented Oct 14, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

lgirdwood commented Oct 14, 2026

Overview

Changes in this PR

1. West Manifest Integration (west.yml)

2. Audio Processing Module (src/audio/ffmpeg_dec)

3. Pipeline & Topology Definitions

Verification & Build Results

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

intel-sofci commented Oct 14, 2026

PR 11308: test results

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

1. West Manifest Integration (`west.yml`)

2. Audio Processing Module (`src/audio/ffmpeg_dec`)