Add audio input support to Responses API - #999
Open
Akshay Sonawane (apsonawane) wants to merge 2 commits into
Open
Add audio input support to Responses API#999Akshay Sonawane (apsonawane) wants to merge 2 commits into
Akshay Sonawane (apsonawane) wants to merge 2 commits into
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Akshay Sonawane (apsonawane)
enabled auto-merge (squash)
August 14, 2026 01:07
Copilot started reviewing on behalf of
Akshay Sonawane (apsonawane)
August 14, 2026 01:08
View session
Akshay Sonawane (apsonawane)
requested review from
Baiju Meswani (baijumeswani) and
Scott McKay (skottmckay)
August 14, 2026 01:10
Contributor
There was a problem hiding this comment.
Pull request overview
Extends the C++ Responses API pipeline to accept audio alongside text and images.
Changes:
- Decodes and validates base64 audio input.
- Adds audio and combined-media prompt processing.
- Calculates token budgets after multimodal expansion and adds unit coverage.
Reviewed changes
Copilot reviewed 6 out of 6 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
response_converter_test.cc |
Tests audio conversion and invalid base64. |
onnx_chat_generator_vision_test.cc |
Tests audio and combined-media markers. |
response_converter.cc |
Converts audio payloads into owned items. |
onnx_chat_generator.h |
Generalizes generator interfaces for media. |
onnx_chat_generator.cc |
Processes image/audio inputs and token sizing. |
chat_session.cc |
Detects and routes audio media turns. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+433
to
+434
| } else if (part.view && part.view->type == FOUNDRY_LOCAL_ITEM_AUDIO) { | ||
| audios.push_back(static_cast<const AudioItem*>(part.view)); |
Comment on lines
+198
to
+201
| if (part.view->type == FOUNDRY_LOCAL_ITEM_IMAGE) { | ||
| content.push_back(nlohmann::json{{"type", "image"}}); | ||
| } else if (part.view->type == FOUNDRY_LOCAL_ITEM_AUDIO) { | ||
| content.push_back(nlohmann::json{{"type", "audio"}}); |
| EXPECT_EQ(bytes[3], 4u); | ||
| } | ||
|
|
||
| TEST(ResponseConverterTest, ToSessionRequest_InputAudio_RejectsInvalidBase64) { |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add audio input support to Responses API
Summary
Extend the SDK v2 Responses API multimodal pipeline to support audio alongside text and images.
Changes
input_audiopayloads into ownedAudioIteminstances.<|audio|>markers to multimodal prompts.ProcessImagesfor imagesProcessAudiosfor audioProcessImagesAndAudiosfor combined mediainput_ids.Motivation
The Responses schema already parsed
input_audio, and the runtime already providedAudioItemand ORT GenAI audio-processing APIs. However, the Responses converter rejected audio before it reached the model.Media expansion also happened after generator sizing, which could produce an insufficient
max_length.