Support chat-style multimodal audio LMs in speech WER evaluation#2571
Merged
Conversation
The genai speech evaluator only routed `whisper` and `nemotron_speech` model types and raised for anything else, so chat-style multimodal LMs with an audio input (e.g. gemma4, and other audio-instruction models) could not be evaluated for WER/RTFx without a custom script. Changes: - Detect chat-style audio LMs generically via a `speech`/`audio` component in `genai_config.json` (`_is_multimodal_lm_genai`, not a hard-coded model type) and route them to a new `_inference_text_genai_multimodal` path. It builds the `<|audio|>` chat prompt from the model's `chat_template.jinja` with a configurable system prompt + instruction (defaulting to a strict ASR prompt that suppresses chat-style refusals), greedy-decodes, and returns only the newly generated tokens as the transcription. The system prompt/instruction can be overridden via the metric's pre-process params. Dedicated ASR heads (whisper/nemotron_speech) are unaffected: they use decoder-token prompts and cannot emit chat-style refusals, so a system prompt does not apply. - Normalize text in `WordErrorRate` by default (lowercase, strip punctuation, collapse whitespace), matching standard ASR WER practice. Previously raw text was passed to torchmetrics, so casing/punctuation the model emits but references omit were counted as errors (e.g. FLEURS references are already normalized). Toggleable via `normalize`. Validated end-to-end: gemma4 int4 on FLEURS en_us reports WER 0.0926, matching the reference standalone script (raw/un-normalized was 0.2171). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 1a5291c2-3155-439c-97cf-4febeab28ac5
Contributor
There was a problem hiding this comment.
Pull request overview
This PR extends Olive’s genai speech evaluator so chat-style multimodal audio LMs (e.g., gemma4-like models) can be evaluated with WER/RTFx, and it updates WER scoring to use standard ASR-style text normalization by default.
Changes:
- Add generic detection for chat-style audio LMs via
genai_config.json(speech/audiocomponent) and route speech evaluation to a new multimodal genai inference path. - Implement multimodal genai ASR inference that builds a chat prompt (system + instruction + audio turn) and returns only generated transcription text.
- Normalize text by default in
WordErrorRate(lowercase, strip punctuation, collapse whitespace) with anormalizetoggle and corresponding unit tests.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
olive/evaluator/olive_evaluator.py |
Adds audio-LM detection and a new genai multimodal inference path for speech evaluation routing. |
olive/evaluator/accuracy.py |
Introduces default WER text normalization with a configurable normalize option. |
test/evaluator/test_olive_evaluator.py |
Adds unit tests for multimodal audio-LM detection and WER normalization behavior. |
The whisper, streaming, and multimodal-LM speech inference methods each duplicated the onnxruntime-genai import guard, genai_config load + og.Model provider build, and the entire per-batch transcribe/timing/ collect loop (only the per-clip transcribe call differed). Extract two shared helpers on OnnxEvaluator: - `_load_genai_speech_model(model, device)` -> (og, og_model, genai_config, model_dir): import guard, genai_config load, provider selection. - `_run_speech_inference_loop(dataloader, sample_rate, transcribe_fn)`: the shared batch loop, parameterized by a `transcribe_fn(audio) -> str`. Each method now keeps only its model-specific transcribe closure. No behavior change; net -85 lines. Re-validated gemma4 FLEURS en_us (WER 0.0926 unchanged); evaluator test suite 125 passed. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 1a5291c2-3155-439c-97cf-4febeab28ac5
…essages - `_is_multimodal_lm_genai`: return False when the genai config's `model` section is not a dict (e.g. a malformed `"model": null`) instead of raising TypeError on the membership check. Added a regression test. - `apply_chat_template`: pass the messages JSON positionally (it is the first parameter of the ORT-GenAI signature) while keeping the keyword-only options as keywords. gemma4 FLEURS en_us re-validated (WER 0.0926 unchanged). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 1a5291c2-3155-439c-97cf-4febeab28ac5
xiaoyu-work
approved these changes
Jul 15, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Describe your changes
The genai speech evaluator (
_evaluate_onnx_accuracy→ text-based/WER path) only routedwhisperandnemotron_speechgenai model types and raisedValueErrorfor anything else. Chat-style multimodal LMs that accept an audio input (e.g.gemma4, and other audio-instruction models) therefore could not be evaluated for WER/RTFx through Olive without a custom script.This PR adds first-class support for those models, plus a correctness fix to WER scoring:
1. Generic chat-style audio-LM inference path
_is_multimodal_lm_genai()detects these models generically by the presence of aspeech/audiocomponent ingenai_config.json— not a hard-coded model type — so any current/future chat-style audio LM is covered._inference_text_genai_multimodal(): builds the<|audio|>chat prompt from the model'schat_template.jinjawith a configurable system prompt + instruction, greedy-decodes, and returns only the newly generated tokens as the transcription.system_promptandinstructionare overridable via the metric's pre-process params.whisper,nemotron_speech) are intentionally unaffected — they use decoder-token prompts and cannot emit chat-style refusals, so a system prompt does not apply to them.2. Standard WER text normalization
WordErrorRate.measurenow normalizes text by default (lowercase, strip punctuation, collapse whitespace), matching standard ASR WER practice. Previously raw strings were passed straight totorchmetrics.WordErrorRate, so casing and punctuation the model emits — but references omit (e.g. FLEURS references are already lowercased/de-punctuated) — were counted as word errors. Toggleable via a newnormalizeconfig option (defaultTrue).Validation
End-to-end
olive runon agemma4int4 export over FLEURSen_us(64 samples) reports WER 0.0926, exactly matching the reference standalone script. Without normalization the same run scored 0.2171, confirming the normalization gap was purely casing/punctuation.Checklist before requesting a review
test/evaluator/test_olive_evaluator.py)test/evaluator/test_olive_evaluator.py: 125 passed)lintrunner -a(ruff check + format clean on changed files)(Optional) Issue link