WIP: Ffmpeg decoder, encoders and filters.#10990
Draft
lgirdwood wants to merge 6 commits into
Draft
Conversation
Add a loadable (LLEXT) audio module that wraps FFmpeg. Two build modes: - decoder (default): a compressed elementary stream (FLAC/AAC/Opus, selected via Kconfig) is parsed and decoded to PCM (process_raw_data); - filter (CONFIG_FFMPEG_DEC_FILTER_MODE): a PCM source/sink effect that runs an FFmpeg audio filter graph (afftdn noise reduction). FFmpeg is a west-pinned source (see west.yml) cross-built by ffmpeg.cmake as a CMake ExternalProject, enabling only the Kconfig-selected decoders and filters. The module supplies the libc surface FFmpeg needs that the SOF core does not export to LLEXT (ffmpeg_dec-shims.c), a SOF-heap-backed malloc (ffmpeg_dec-alloc.c), fast single-precision float math (fastmathf.c), and routes av_log() into the Zephyr log. The avfilter graph backend and the PCM effect ops are in ffmpeg_dec-filter.c. Also adds the rimage module manifest (ffmpeg_dec.toml + per-platform includes), the topology widget (ffmpeg_dec.conf), host test tooling (ffmpeg_dec_prepare.sh) and docs (README/TESTING). Verified building and load-ready (all symbols resolved, signed) for ace30 (ptl) with the Zephyr SDK: FLAC and FLAC+AAC decoders, and the afftdn filter effect. Runtime decode/denoise verification is pending hardware. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add the ffmpeg_dec component to the host->component->host benchmark harness so a topology instantiating the module can be built and driven (testbench or target). Adds the per-format bench configs (generated with bench_comp_generate.sh), registers the widget include and the ffmpeg_dec32 BENCH_CONFIG in cavs-benchmark-hda.conf, and adds ffmpeg_dec to the s32 component list in tplg-targets-bench.cmake (the filter effect path processes S32). Also drop the codec-setup bytes control from the ffmpeg_dec widget class so it is a plain 1-in/1-out effect; the decoder's STREAMINFO control is added at the instance level in a decoder topology instead. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add CONFIG_FFMPEG_DEC_MP3: enables the libavcodec mp3 decoder and mpegaudio parser in the cross-build, the FFMPEG_DEC_CODEC_MP3 codec id mapping, and the float math layer (the mp3 decoder uses it). Verified building for ace30 (ptl) with decoders [flac,mp3]. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
FFmpeg has no native MP3 encoder, so add MP3 encode through libshine, a small fixed-point encoder (good fit for a DSP - no float dependency). libshine is a west-pinned project; ffmpeg.cmake cross-builds it (its lib needs no config.h and its autotools CLI/shared link cannot work bare-metal, so the objects are compiled and archived directly), generates a shine.pc, and enables --enable-libshine --enable-encoder=libshine in FFmpeg. FFmpeg's require_pkg_config link test for -lshine pulls newlib malloc and hence Zephyr-runtime symbols (z_errno_wrap, ...) that only exist at module load; a configure-test-only stub is passed via --extra-ldflags so the test links (--extra-ldflags does not affect the static-archive build). Gated by CONFIG_FFMPEG_ENC_MP3. Verified building for ace30 (ptl): libshine.a cross-builds and libavcodec.a contains the libshine encoder. A module encode path (PCM->MP3) is a separate build mode. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add CONFIG_FFMPEG_DEC_ENCODE_MODE: build the module as an encoder using the modern .process_raw_data path in reverse of the decoder - avcodec_send_frame/avcodec_receive_packet. ffmpeg_dec-encode.c opens the libshine MP3 encoder, converts SOF interleaved S32 to the encoder sample format per frame, and emits the compressed elementary stream. The interface ladder in ffmpeg_dec.c selects encoder / filter / decoder ops by Kconfig. libshine is linked into the module (after libavcodec, which references it) from its own cross-build install dir. ffmpeg.cmake now allows an encoder-only build (no decoder) and makes the decoder configure flags conditional. The encoder path pulls a few more libc symbols, added to the local surface: calloc/posix_memalign (alloc), modf/localtime_r/iconv*/__xpg_strerror_r (shims). fastmathf is now always built for the real backend (its sofm_log2f backs the shims' log10()). Verified building and load-ready (no unresolved externals, signed) for ace30 (ptl) as an encoder-only module. Runtime encode + real-time framing need hardware. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add HIFI.md: analysis of the hot audio-processing paths in the module (decode/ filter/encode) and the linked FFmpeg DSP, and where Xtensa HiFi intrinsics would help. FFmpeg is built --disable-asm (scalar C on Xtensa) and has no _xtensa DSP init, so the generic C kernels run. Documents: the module's own PCM conversion loops (easy HiFi wins we own), FFmpeg's DSP dispatch contexts (float_dsp, flacdsp, mpegaudiodsp, tx) as fork-patch targets mirroring the other arch inits, fastmathf vectorisation, and a cost/benefit priority order. References SOF's existing HiFi kernels (src/math/*_hifi*) as the template. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
WIP build support for llext based and runtime linked ffmpeg audio encoding, decoding and filters as a west module. Builds and links today. Next steps are integration with topology and kernel then optimization.