qwen3_asr: support larger output#106
Open
jl wants to merge 5 commits into
Open
Conversation
The qwen3_asr model supports over an hour of input audio. Remove the hard-coded 256 output token limit, which practically constrained input audio to around 1-2 minutes before running into output truncated error. Also remove duplicate code in several places.
So that run() and run_batch() are more parallel.
Contributor
|
Have you considered #74? |
Author
|
#74 looks nice! After that lands, then the scope of this PR would be reduced to providing convenient defaults even if |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Problem: qwen3_asr model would truncate output on medium-length audio clips (anything longer than a minute, or 256 output tokens).
The original code had a hard-coded limit of 256 output tokens:
Even with the (very large) default context_size of 65536, the maximum number of generated output tokens was still limited to 256 (representing about a minute of spoken transcription text) even while the input could support thousands of minutes of input audio. Because of the fixed amount, the context allocation was effectively (99.7% audio, 0.3% output). This PR changes the context allocation strategy to a variable amount based on
--n-ctxor the model default (75% audio, 25% output tokens).Scope
The only file changed is
arch/qwen3_asr/model.cpp, and the behavior is functionally the same, except that now it won't truncate generated output on long audio inputs. When --n-ctx is not specified, memory usage will increase for short audio inputs, as the output token allocation will be large.Future work may consider a command line parameter to explicit set a gen_reserve override instead of automatically allocating 25% of the available context. This could be useful in edge cases of very memory-constrained environments that need precise control over audio/generated token allocation, or long recordings of someone speaking extremely quickly.
AI Assistance
No AI assistance was used to write, test, or comment the code.
Validation
There should not be any change to WER or numerical validation as a result of these changes.