WIP: Ffmpeg decoder, encoders and filters.

src/audio/CMakeLists.txt

@@ -41,6 +41,9 @@ if(NOT CONFIG_COMP_MODULE_SHARED_LIBRARY_BUILD)
 	if(CONFIG_COMP_DRC)
 		add_subdirectory(drc)
 	endif()
+	if(CONFIG_COMP_FFMPEG_DEC)
+		add_subdirectory(ffmpeg_dec)
+	endif()
 	if(CONFIG_COMP_FIR)
 		add_subdirectory(eq_fir)
 	endif()

src/audio/Kconfig

@@ -139,6 +139,7 @@ rsource "dcblock/Kconfig"
 rsource "drc/Kconfig"
 rsource "eq_fir/Kconfig"
 rsource "eq_iir/Kconfig"
+rsource "ffmpeg_dec/Kconfig"
 rsource "google/Kconfig"
 rsource "igo_nr/Kconfig"
 rsource "mfcc/Kconfig"

src/audio/ffmpeg_dec/CMakeLists.txt

@@ -0,0 +1,18 @@
+# SPDX-License-Identifier: BSD-3-Clause
+
+if(CONFIG_COMP_FFMPEG_DEC STREQUAL "m" AND DEFINED CONFIG_LLEXT)
+  add_subdirectory(llext ${PROJECT_BINARY_DIR}/ffmpeg_dec_llext)
+  add_dependencies(app ffmpeg_dec)
+else()
+  add_local_sources(sof ffmpeg_dec.c)
+
+  if(CONFIG_COMP_FFMPEG_DEC_STUB)
+    add_local_sources(sof ffmpeg_dec-stub.c)
+  else()
+    add_local_sources(sof ffmpeg_dec-ffmpeg.c)
+    # NOTE: ffmpeg_dec-shims.c / ffmpeg_dec-alloc.c are LLEXT-only (see
+    # llext/CMakeLists.txt). They provide libc/libm symbols the isolated LLEXT
+    # cannot import; in a built-in (=y) or native/testbench build the real libc
+    # provides those, so including them here would multiply-define malloc etc.
+  endif()
+endif()

src/audio/ffmpeg_dec/HIFI.md

@@ -0,0 +1,152 @@
+# HiFi intrinsics opportunities in ffmpeg_dec
+
+Analysis of the hot audio-processing paths in the FFmpeg SOF module (decode /
+filter / encode) and where Xtensa **HiFi** SIMD intrinsics would give the biggest
+wins. It covers both the code this module owns and the FFmpeg code it links.
+
+## Background: why this matters on Xtensa
+
+- The FFmpeg archive is cross-built with **`--disable-asm`** (see `ffmpeg.cmake`),
+  because FFmpeg has no Xtensa assembly. Every DSP kernel therefore runs as
+  **scalar C** on the DSP — no SIMD, no fused multiply-add vectorisation.
+- FFmpeg dispatches its DSP through function-pointer contexts with per-architecture
+  init hooks (`ff_*_init_aarch64/arm/x86/riscv`) — but **no `_xtensa`** variant. The
+  generic `_c` kernels are what execute on our target.
+- SOF already ships HiFi-optimised DSP and the toolchain for it. Use it as the
+  template:
+  - ISA select + intrinsics header (`src/math/exp_fcn_hifi.c`):
+    ```c
+    #if   XCHAL_HAVE_HIFI5
+    #include <xtensa/tie/xt_hifi5.h>
+    #elif XCHAL_HAVE_HIFI4
+    #include <xtensa/tie/xt_hifi4.h>
+    #else
+    #include <xtensa/tie/xt_hifi3.h>
+    #endif
+    ```
+  - Existing HiFi kernels to mirror: `src/math/fir_hifi{2ep,3,5}.c`,
+    `src/math/iir_df1_hifi{3,4,5}.c`, `src/math/fft/fft_{16,32}_hifi3.c`,
+    `src/math/exp_fcn_hifi.c`. `ace30` (ptl) is HiFi4-class.
+
+There are two distinct bodies of code:
+
+- **(A) Module glue we own** — the PCM format-conversion loops in
+  `ffmpeg_dec-*.c`. Small, per-sample, easy to hand-vectorise with HiFi, and we
+  can change them freely.
+- **(B) FFmpeg DSP** — the actual codec compute (MDCT/FFT, LPC, windowing). Much
+  higher cycle cost, but upstream; adding HiFi means either FFmpeg fork patches
+  (an `_xtensa` DSP init, mirroring the other arch dirs) or routing to SOF's own
+  HiFi kernels.
+
+## Hot-path summary
+
+| # | Path | File | Kind | Now | HiFi win | Effort |
+|---|------|------|------|-----|----------|--------|
+| 1 | MDCT/FFT (AAC/MP3/Opus/afftdn) | FFmpeg `libavutil/tx*` | float/fixed transform | scalar C | **very high** | high |
+| 2 | float vector kernels (fmul/window/overlap-add) | FFmpeg `libavutil/float_dsp` | float SIMD | scalar C | **high** | low–med |
+| 3 | FLAC LPC / residual | FFmpeg `libavcodec/flacdsp` | int32/64 MAC | scalar C | high | med |
+| 4 | MP3 synthesis window / imdct36 | FFmpeg `libavcodec/mpegaudiodsp` | float/fixed | scalar C | high | med |
+| 5 | AAC PS / SBR DSP | FFmpeg `aacpsdsp`/`sbrdsp` | float SIMD | scalar C | med (HE-AAC only) | med |
+| 6 | S32↔float PCM convert (filter) | `ffmpeg_dec-filter.c:230,255` | int↔float + scale | scalar C | med | **low** |
+| 7 | planar→interleaved PCM (decode) | `ffmpeg_dec-ffmpeg.c:231` | copy/interleave | scalar C | low–med | **low** |
+| 8 | S32→S16 pack (encode) | `ffmpeg_dec-encode.c:147` | narrow + interleave | scalar C | med | **low** |
+| 9 | fast float libm (powf/exp2f/...) | `fastmathf.c` | scalar poly/Newton | scalar C | med (per-band) | med |
+
+Priority order (cost/benefit): **2 → 6/7/8 → 3 → 1 → 4 → 9 → 5**. Start with the
+easy, high-use float vector kernels (2) and the conversion loops we own (6–8);
+tackle the transform (1) last because it is the most work despite the highest
+raw cost.
+
+## (A) Module conversion loops — we own these, do them first
+
+These run **per sample over a whole frame** (typically 1024–4608 samples ×
+channels per call), so they are genuinely hot and are trivial, self-contained HiFi
+wins. Add HiFi versions guarded by `#if XCHAL_HAVE_HIFI*` with the current scalar
+loop as the `#else` fallback.
+
+1. **Filter S32→float deinterleave + normalise** — `ffmpeg_dec-filter.c:230`
+   ```c
+   ((float *)in->data[c])[i] = (float)src[i * ch + c] / FFMPEG_AF_S32_SCALE;
+   ```
+   HiFi: load 32-bit lanes, convert int32→float (`AE_FLOAT32`-family), multiply by
+   the reciprocal scale, store to the per-channel plane. Deinterleave with
+   strided/`AE_SEL` moves.
+
+2. **Filter float→S32 interleave + denormalise + clip** — `ffmpeg_dec-filter.c:255`
+   The scalar path multiplies, branches to clamp, then casts. HiFi does the
+   float→int32 convert **with saturation** in one op (no per-sample branch), which
+   is both faster and removes the mispredicted clamp branch.
+
+3. **Encode S32→S16 narrow + deinterleave** — `ffmpeg_dec-encode.c:147`
+   ```c
+   int16_t s = (int16_t)(in[i * ch + c] >> 16);
+   ```
+   HiFi: pack 32→16 with rounding (`AE_ROUND16X4F32`-style) and store; SIMD handles
+   4–8 samples per iteration.
+
+4. **Decode planar→interleaved** — `ffmpeg_dec-ffmpeg.c:231`
+   Per-element `memcpy`-equivalent; a HiFi interleave (vector load per plane,
+   `AE_SEL` interleave, vector store) removes the per-sample loop overhead. Lower
+   priority as many decoders already output the sink format.
+
+A single small `ffmpeg_dec-convert.c` with HiFi + scalar variants (like SOF's
+`*_generic.c` / `*_hifi3.c` split) would hold all four and be reusable across the
+three modes.
+
+## (B) FFmpeg DSP — the real compute
+
+FFmpeg's DSP contexts are function-pointer tables initialised per arch. The
+Xtensa-idiomatic fix is a **fork patch** adding `ff_<dsp>_init_xtensa()` with HiFi
+intrinsics, mirroring `libavcodec/aarch64/`, `x86/` etc. (our FFmpeg is already a
+SOF fork — see `west.yml` — so such patches have a home).
+
+- **`AVFloatDSPContext`** (`libavutil/float_dsp.h`) — `vector_fmul`,
+  `vector_fmul_window`, `vector_fmul_add`, `vector_fmul_reverse`,
+  `scalarproduct_float`. Used for windowing / overlap-add by **AAC, MP3 and Opus**.
+  These are plain multiply/MAC over contiguous float arrays — the easiest and
+  highest-leverage HiFi target (one small `float_dsp_init_xtensa` benefits three
+  codecs). **Do this first on the FFmpeg side.**
+- **`FLACDSPContext`** (`libavcodec/flacdsp.h`) — `lpc16`/`lpc32`/`lpc33` and
+  decorrelate. FLAC decode is dominated by the LPC prediction MAC loop over the
+  residual: a natural fit for HiFi multiply-accumulate. `ff_flacdsp_init_xtensa`.
+- **`MPADSPContext`** (`libavcodec/mpegaudiodsp.h`) — `apply_window_{float,fixed}`,
+  `imdct36_blocks_*`, `synth_filter`. The MP3 subband synthesis window is the MP3
+  decode hot loop.
+- **`libavutil/tx`** (MDCT/FFT) — the single largest cost for AAC/MP3/Opus decode
+  and for the `afftdn` filter. It is a "codelet" system rather than one function
+  pointer, so it is the most work. Two routes:
+  1. Add HiFi FFT/MDCT codelets to the fork (largest effort, cleanest for FFmpeg).
+  2. Bridge to **SOF's existing HiFi3 FFT** (`src/math/fft/fft_*_hifi3.c`) by
+     replacing FFmpeg's transform behind `av_tx_init` for the sizes SOF supports.
+     Less code, but a semantic bridge (twiddle/scaling/layout must match) and only
+     covers power-of-two sizes.
+- **`PSDSPContext` / SBR DSP** — only relevant if HE-AAC (SBR/PS) is enabled; skip
+  unless needed.
+
+## (C) fastmathf — the module's float libm
+
+`fastmathf.c` (`powf`/`exp2f`/`log2f`/`sinf`/`cosf`/`sqrtf`/`cbrtf`) is scalar
+polynomial/Newton code. These are called per-band / per-frame (not the innermost
+per-sample loop), so they are a secondary target. HiFi wins come from:
+- `sqrtf`/`cbrtf`: HiFi `RSQRT`/reciprocal seed instructions instead of the
+  bit-trick + Newton.
+- `exp2f`/`log2f`/`sinf`/`cosf`: process 2/4 lanes at once when a decoder needs a
+  vector of them (e.g. AAC scalefactor gains), and use FMA for the polynomials.
+  Mirror `src/math/exp_fcn_hifi.c`.
+
+## Recommendations
+
+1. **Quick wins we own** — hand-vectorise the four conversion loops (§A) with
+   `#if XCHAL_HAVE_HIFI*` + scalar fallback. Small, self-contained, no fork needed.
+2. **Biggest FFmpeg leverage-per-effort** — add `ff_float_dsp_init_xtensa` (HiFi
+   `vector_fmul*`/`scalarproduct`) to the FFmpeg fork; benefits AAC + MP3 + Opus.
+3. **Per-codec kernels** — `flacdsp` (FLAC), `mpegaudiodsp` (MP3) HiFi inits as
+   fork patches, gated on the enabled decoders.
+4. **Transform** — evaluate SOF-HiFi-FFT bridge vs. HiFi tx codelets for
+   `libavutil/tx`; highest payoff, most work, do once the cheaper items land.
+5. Always keep the scalar C path as the fallback (`#else`) and behind the same ISA
+   guards SOF uses, so non-HiFi / host/testbench builds still work.
+
+Measurement: profile on hardware per codec before and after; the ranking above is
+by expected cost, but the actual hot spot depends on the codec and content (FLAC
+is LPC-bound, AAC/MP3/Opus are transform+window-bound).

src/audio/ffmpeg_dec/Kconfig

@@ -0,0 +1,122 @@
+# SPDX-License-Identifier: BSD-3-Clause
+
+config COMP_FFMPEG_DEC
+	tristate "FFmpeg (libavcodec) audio decoder"
+	select COMP_FFMPEG_DEC_STUB if COMP_STUBS
+	help
+	  Select to include the FFmpeg audio decoder module. It wraps
+	  libavcodec decoders behind the SOF module interface, decoding a
+	  compressed elementary stream to PCM. The real backend cross-builds
+	  libavcodec/libavutil/libswresample from the FFmpeg source pulled in
+	  by west (see west.yml), enabling only the decoders selected below.
+	  Without the source (or for CI) select COMP_FFMPEG_DEC_STUB to build
+	  the dependency-free passthrough stub.
+
+config COMP_FFMPEG_DEC_STUB
+	bool "FFmpeg decoder stub backend"
+	depends on COMP_FFMPEG_DEC
+	help
+	  Build the ffmpeg_dec module against a dependency-free passthrough
+	  backend instead of libavcodec. Intended for testing and CI so the
+	  SOF glue and LLEXT packaging can be validated without the FFmpeg
+	  source or a cross-built archive.
+
+if COMP_FFMPEG_DEC && !COMP_FFMPEG_DEC_STUB
+
+comment "FFmpeg decoders to build (footprint scales with selection)"
+
+config FFMPEG_DEC_FLAC
+	bool "FLAC decoder"
+	default y
+	help
+	  Lossless integer decoder. Smallest footprint, no floating-point
+	  math. Enables --enable-decoder=flac in the libavcodec build.
+
+config FFMPEG_DEC_AAC
+	bool "AAC-LC decoder"
+	help
+	  AAC Low Complexity decoder. Pulls in the single-precision float
+	  math layer (see FFMPEG_DEC_FLOAT_MATH). Enables
+	  --enable-decoder=aac in the libavcodec build.
+
+config FFMPEG_DEC_OPUS
+	bool "Opus decoder"
+	help
+	  Opus (SILK+CELT) decoder. Pulls in the single-precision float math
+	  layer. Enables --enable-decoder=opus in the libavcodec build.
+
+config FFMPEG_DEC_MP3
+	bool "MP3 decoder"
+	help
+	  MPEG-1/2 Layer III decoder. Enables --enable-decoder=mp3 and the
+	  mpegaudio parser in the libavcodec build. Uses the float math layer.
+
+config FFMPEG_ENC_MP3
+	bool "MP3 encoder (libshine, fixed-point)"
+	help
+	  MPEG-1 Layer III encoder via libshine, a small fixed-point encoder
+	  (west-pinned; FFmpeg has no native MP3 encoder). Cross-builds libshine
+	  and enables --enable-libshine --enable-encoder=libshine. Fixed-point, so
+	  no float math dependency. The encoder is built into the archive; a module
+	  encode path (PCM->MP3) is a separate build mode.
+
+comment "FFmpeg audio filters (libavfilter; need the avfilter-graph backend to run)"
+
+config FFMPEG_FILTER_AFFTDN
+	bool "FFT noise reduction (afftdn)"
+	help
+	  Build FFmpeg's FFT-based denoiser (afftdn) into libavfilter. Pure DSP,
+	  no external model file (unlike arnndn). Enabling any filter turns on
+	  libavfilter in the cross-build. NOTE: a filter is only usable at runtime
+	  once the module gains an avfilter-graph backend; selecting it here just
+	  builds the filter into the archive.
+
+config FFMPEG_DEC_ENCODE_MODE
+	bool "Build as an audio encoder (PCM -> compressed) instead of a decoder [EXPERIMENTAL]"
+	select FFMPEG_ENC_MP3
+	help
+	  Build the module as an encoder: PCM in, compressed elementary stream out
+	  (MP3 via libshine), driven with avcodec_send_frame/avcodec_receive_packet
+	  (the reverse of the decoder). Selects the libshine MP3 encoder. Do not set
+	  together with FFMPEG_DEC_FILTER_MODE. Structurally complete and load-ready;
+	  real-time framing (MP3 fixed 1152-sample frames) needs on-hardware tuning.
+
+config FFMPEG_DEC_FILTER_MODE
+	bool "Build as a PCM filter effect instead of a decoder [EXPERIMENTAL]"
+	select FFMPEG_FILTER_AFFTDN
+	help
+	  Build the module as a PCM source->sink effect that runs an FFmpeg
+	  audio filter graph (default afftdn noise reduction) via the modern
+	  .process interface, instead of the compressed-stream decoder. Turns on
+	  libavfilter and the fast float math. Structurally complete and
+	  load-ready; real-time latency/format tuning needs on-hardware bring-up.
+
+# Turns on libavfilter in the FFmpeg cross-build when any filter is selected.
+config FFMPEG_BUILD_AVFILTER
+	bool
+	default y if FFMPEG_FILTER_AFFTDN
+
+# Selected automatically when a decoder or filter that needs IEEE float math is
+# built. Gates compilation of the fast single-precision libm (fastmathf.c).
+config FFMPEG_DEC_FLOAT_MATH
+	bool
+	default y if FFMPEG_DEC_AAC || FFMPEG_DEC_OPUS || FFMPEG_DEC_MP3 || FFMPEG_FILTER_AFFTDN
+
+config FFMPEG_DEC_COLD_SPLIT
+	bool "Place cold FFmpeg code (av_cold init/setup) in DRAM [EXPERIMENTAL]"
+	depends on COLD_STORE_EXECUTE_DRAM
+	default n
+	help
+	  EXPERIMENTAL. Build libavcodec with -mtext-section-literals and rename
+	  FFmpeg's cold functions (its av_cold init/table-generation code, which GCC
+	  emits into .text.unlikely) into SOF's .cold section so the LLEXT loader
+	  places them in DRAM, keeping the hot per-frame decode in fast SRAM.
+
+	  Measured cold code is small (~1 KB for FLAC, ~26 KB for AAC) because
+	  FFmpeg only marks init functions av_cold. Placing that much cold code in a
+	  -shared LLEXT currently hits Xtensa linker limits (cold->hot call8 range
+	  needs -mlongcalls; the orphan .cold lands at a bad LMA and overlaps .hash),
+	  so this is off by default pending proper cold-region placement in the LLEXT
+	  memory map. See git history / TESTING notes.
+
+endif # COMP_FFMPEG_DEC && !COMP_FFMPEG_DEC_STUB

src/audio/ffmpeg_dec/README.md

@@ -0,0 +1,59 @@
+# FFmpeg audio decoder module (ffmpeg_dec)
+
+Wraps FFmpeg's `libavcodec` audio decoders behind the SOF `module_interface`,
+decoding a compressed elementary stream to PCM inside the DSP. First target
+codec is **FLAC**.
+
+## Design
+
+The SOF glue and the decoder are separated so the integration can be validated
+without libavcodec:
+
+- `ffmpeg_dec.c` — SOF module core: `init` / `prepare` / `process_raw_data` /
+  `set_configuration` / `reset` / `free`, plus the LLEXT manifest. Input is the
+  compressed byte stream (raw data), output is interleaved PCM. Decoding is
+  delegated to a `struct ffmpeg_dec_backend`.
+- `ffmpeg_dec-stub.c` — dependency-free passthrough backend. Lets CI build and
+  exercise the module (pipeline, IPC config, LLEXT packaging) with **no** FFmpeg
+  libraries. Selected by `CONFIG_COMP_FFMPEG_DEC_STUB`.
+- `ffmpeg_dec-ffmpeg.c` — real backend driving the standalone `libavcodec`
+  send-packet / receive-frame API. A raw stream is framed on-DSP with an
+  `AVCodecParser` (`av_parser_parse2`); decode is forced single-threaded
+  (`thread_count = 1`). Requires pre-compiled static libraries and headers under
+  `third_party/` (see below).
+- `ffmpeg_dec.h` — private data (`struct ffmpeg_dec_comp_data`) and the backend
+  interface.
+
+The codec setup header (e.g. FLAC STREAMINFO) is delivered as a binary control
+via `set_configuration` and stored as `extradata`, which the libavcodec backend
+installs before `avcodec_open2()`.
+
+## Build
+
+- **Stub (default for testing/CI):** `CONFIG_COMP_FFMPEG_DEC=y` (or `=m` for
+  LLEXT) with `CONFIG_COMP_FFMPEG_DEC_STUB=y`. No external dependencies.
+- **Real decoder:** `CONFIG_COMP_FFMPEG_DEC=m`, `CONFIG_COMP_FFMPEG_DEC_STUB=n`.
+  Requires cross-compiled decoder-only FFmpeg static libraries installed as:
+  - `third_party/lib/libavcodec.a`, `libavutil.a`, `libswresample.a`
+  - `third_party/include/libavcodec/…`, `libavutil/…`, `libswresample/…`
+  Configure FFmpeg with `--disable-everything --disable-avformat
+  --disable-pthreads --enable-decoder=flac --enable-parser=flac` (plus the
+  target cross-compile flags).
+
+## Files
+
+- `Kconfig` — `COMP_FFMPEG_DEC` (tristate) and `COMP_FFMPEG_DEC_STUB`.
+- `CMakeLists.txt` / `llext/CMakeLists.txt` — static and LLEXT builds; the LLEXT
+  target name is `ffmpeg_dec` and links the FFmpeg libraries in the non-stub
+  branch.
+- `ffmpeg_dec.toml` / `llext/llext.toml.h` — rimage module manifest
+  (`UUIDREG_STR_FFMPEG_DEC`); OBS is sized above IBS since PCM out ≫ compressed
+  in.
+
+## Status / TODO
+
+- Codec id is currently hard-coded to FLAC in `init`; wire it from topology/IPC
+  init config for other codecs.
+- Output is assumed to already match the sink sample format; add
+  `libswresample` (or a fixed-point path) for format/rate conversion.
+- Topology and host test tooling for end-to-end (bit-exact) FLAC decode.