feat(media): give audio and Slack video a URL the agent can actually fetch - #1460
feat(media): give audio and Slack video a URL the agent can actually fetch#1460ShinyChang wants to merge 40 commits into
Conversation
Audio was the only inbound file class the agent could never act on. With STT enabled it received a transcript and nothing else; with STT disabled it received nothing at all, since Discord and Slack added a reaction and dropped the file while the gateway logged at debug and dropped it. Image, video, text and binary attachments all already have a passthrough. Emit an [Audio attachment] block carrying filename, content type and size on all three inbound paths, independent of the STT setting, so a transcript augments the file rather than replacing it. Discord and Slack attach a filestore presigned URL when a filestore is configured, and otherwise fall back to the platform URL with a note naming its access requirement. The gateway holds raw bytes rather than a fetchable location, so it uploads them to the filestore when one exists and emits metadata only when none does. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NhMdtp4VVrqozRfx8uWxjz
Slack special-cased video inside the NotAnImage arm to emit url_private_download directly, bypassing the filestore path its sibling branch uses for PDF and ZIP. That URL needs an Authorization: Bearer header the agent does not hold, so the block pointed at a link that always 403s, with no note explaining why. Route it through filestore like every other non-image attachment, and fall back to the platform URL with a note naming the credential it requires. Extract the block builder into media::video_attachment_block, replacing a private copy in discord.rs and an inline format! in slack.rs that had drifted apart, and share the filename and MIME sanitiser with the audio builder. Discord passes None for the note, so its output is unchanged; a test asserts the exact string rather than substrings to keep it that way. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NhMdtp4VVrqozRfx8uWxjz
The lockfile change had no corresponding Cargo.toml change in this branch. crates/openab-mcp/Cargo.toml already declares http-body-util and tower, so the committed lockfile is simply stale against its own manifest on main and any cargo invocation regenerates those two entries. That staleness is not this PR's to carry. Restored to the base lockfile. Safe for CI: the workflows covering crates/** run no --locked, and the only --locked in CI targets crates/platform-schema's own manifest. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NhMdtp4VVrqozRfx8uWxjz
The two presign helpers returned a bare URL, so all five call sites across discord.rs, slack.rs and gateway.rs rebuilt the same "presigned URL, expires in N minutes" wording themselves. That is the drift this branch removes elsewhere by sharing the block builders, so leaving five copies of the label was inconsistent with its own argument. Both helpers now return the URL paired with its note, built by one private presigned_note(). The call sites collapse to the bare await. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NhMdtp4VVrqozRfx8uWxjz
This comment has been minimized.
This comment has been minimized.
|
Reviewed at 🔴 BlockersNone. The change is well-scoped and the description is unusually honest about its own residual risks, which made verification easy. Nothing below should hold the merge. 🟡 NitsOrdered by what I'd actually act on. Line numbers are at 1. Duplicate failure block on the gateway read-failure path
When bytes fail to read and STT is enabled, two blocks are pushed that say the same thing: This is specific to the read-failed branch — in the transcription-failed case the two blocks are genuinely complementary (the file arrived, transcription didn't work), so the pairing there is correct. Here they're redundant, and the agent has to reconcile two failure signals for one event. Suggest dropping the legacy string on this branch only, in both functions. 2. Gateway audio is uploaded to filestore with the wrong content type
The Discord/Slack path doesn't have this problem: it goes through The bytes are correct and 3. The gateway match arms have no test coverage, in a config CI never testsAll the new tests target Worth pairing with a CI gap: 4. No per-message cap on audio attachmentsText files are capped ( 5. "Always emitted" is conditional on Slack returning a private URL
let url = slack_file_download_url(file);
if url.is_empty() {
continue;
}This guard predates the PR and sits before the 6. On the four unsanitised
|
upload_bytes_and_presign routed the gateway's audio bytes through Filestore::upload_and_presign, whose only previous caller was the text-file path. That method hardcodes text/plain; charset=utf-8 and takes no content_type argument, so the same .m4a was served as audio/mp4 from Slack (which goes through stream_upload_and_presign, honouring the caller's MIME) and as text/plain from Telegram, Feishu, LINE and Google Chat. upload_and_presign now takes content_type and defaults to application/octet-stream, matching the streaming path. The text caller passes its previous value explicitly, so its output is unchanged. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NhMdtp4VVrqozRfx8uWxjz
Discord and Slack inserted the transcript at index 0 of extra_blocks from inside the loop over attachments, which was harmless while a transcript was the only block an audio attachment produced. Now that every attachment also emits a metadata block, three voice notes render as [t3, t2, t1, m1, m2, m3]: the transcripts are reversed and detached from the files they describe, so the agent cannot tell which transcript belongs to which attachment. The gateway had the opposite order again, metadata before transcript. media::audio_attachment_blocks now owns the order and all three adapters extend from it, so the pairing cannot drift per adapter. Order is transcript then metadata, which leaves Discord and Slack byte-identical for the single-attachment case; only the cases that were already wrong change. The gateway's read-failure branch pushed both a metadata block carrying "attachment bytes unavailable (read failed)" and a legacy [Voice message - read failed for <filename>] line, making the agent reconcile two signals for one event. The legacy line is dropped there; the transcription-failure branch keeps its pairing, because there the file did arrive. Its remaining line drops the filename, which the adjacent metadata block already carries sanitised, so the last two unsanitised filename interpolations are gone. process_gateway_event also gains the STT-failure warn! that run_gateway_adapter already had. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NhMdtp4VVrqozRfx8uWxjz
presigned_ttl was capped at 7 days but had no lower bound, and three independent sites render it as ttl_secs / 60: presigned_note, the any-file hint in media.rs, and format_filestore_hint. A configured value below 60 therefore told the agent the URL "expires in 0 minutes" at all three, and the URL would in practice expire before the agent could fetch it. The bound is enforced once instead of fixing three renderings. The existing 7-day cap moves into the same clamp_presigned_ttl helper so both bounds warn symmetrically, and extracting it from Filestore::new makes it testable without an S3 client. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NhMdtp4VVrqozRfx8uWxjz
Two limits raised in review that are behaviour worth writing down rather than code worth changing. Audio has no per-message count cap of its own, unlike text files. The text cap protects the prompt from inlined content and is bypassed once a filestore takes over the upload, and audio bytes are never inlined, so the bound is the platform's own per-message file limit, 10 on both Slack and Discord. Slack forwards an attachment only when its file JSON carries url_private_download or url_private. When both are absent the attachment is skipped before its type is examined, audio included, so "always emitted" holds on Slack only where Slack returned a private URL. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NhMdtp4VVrqozRfx8uWxjz
|
Important CHANGES REQUESTED What This PR DoesThis PR makes Discord, Slack, and gateway audio attachments available as metadata blocks independently of STT. It also routes Slack video through filestore when configured, so the agent can receive a credential-free presigned URL rather than an unusable Slack private URL. How It WorksShared media helpers build sanitized audio and video blocks and pair presigned URLs with their expiry notes. Discord and Slack stream URL-backed media to filestore; gateway adapters upload bytes they already hold. The patch also preserves transcript-to-file ordering and passes the real MIME type to the gateway single-PUT path. Findings
Finding Details🟡 F1: Classify audio by MIME or recognized filename extension
Requested change: add an audio classifier that accepts the platform MIME or a bounded list of recognized audio extensions, use it in both adapters, and test empty and 🟡 F2: Make the storage documentation match the implemented matrixThe video table says Discord with filestore returns a presigned S3 URL, but Requested change: document the current implementation consistently: Discord video remains CDN-only; Slack video and audio use filestore when configured; gateway audio uses the buffered single-PUT path; and the relevant fallback behavior remains explicit. 🟡 F3: Bound aggregate audio work per inbound messageThe loops await every audio attachment and apply no audio-specific count or aggregate-byte budget. The configured filestore default permits 250 MB per file (up to 500 MB), each streaming attempt has a 600-second timeout, and the new docs state that Slack and Discord may supply ten files per message. A single allowed message can therefore schedule multi-gigabyte transfer and serially hold the handler for many timeout windows. STT plus filestore also performs a second download by design. Requested change: add a small audio attachment count limit and aggregate transfer budget before starting downloads, with a clear degraded block or log for skipped items. Cover the cap and a multi-attachment message in tests. 🟡 F4: Cover routing rather than only formatting helpersThe new tests invoke Requested change: introduce a testable routing seam or focused adapter tests. At minimum cover Discord and Slack MIME fallback plus both gateway entry points for STT on/off and filestore/read-failure outcomes. 🟢 F5: Good shared safety and content-type handlingThe shared builders remove control characters from user-controlled metadata before it becomes prompt text. The current head also threads gateway audio MIME to Baseline Check
Addressing External Reviewer Feedback@dogzzdogzz
Reviewer Aggregation
5. Three Reasons We Might Not Need This PR
What's Good (🟢)
|
is_audio_mime was only mime.starts_with("audio/"), while the adjacent
is_video_file already fell back to the filename extension. Discord hands over an
empty string when content_type is absent and Slack does the same for mimetype,
and a CDN commonly labels an upload application/octet-stream, so an ordinary
clip.ogg or meeting.m4a missed the audio branch entirely: without a filestore it
was dropped, with one it became a generic [File: ...] block.
The fallback returns a MIME rather than a bool because a bool would have fixed
only half of it. stt::transcribe builds its multipart body with
Part::mime_str(mime_type).ok()?, which discards the request when the value does
not parse, so admitting an attachment whose MIME is "" would have traded
"audio silently dropped" for "audio always fails to transcribe". A synthesised
type from the extension is what makes the rescued attachment usable.
The extension list deliberately omits webm, mp4 and ogv, the containers that
carry either stream, so this never claims an attachment is_video_file should
handle.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NhMdtp4VVrqozRfx8uWxjz
The gateway builds its audio blocks twice, once in run_gateway_adapter and once in process_gateway_event, and the two copies had already drifted: the earlier duplicate read-failure block and the missing STT-failure warn! both existed on one side only. Reviewers noted that nothing exercises either call site, so a fallback corrected on one path can silently stay wrong on the other. media::audio_blocks_for takes an AudioOutcome (Stored / NoStore / ReadFailed) and owns the mapping from outcome to url and note. Both entry points now pass an outcome instead of assembling the arguments themselves, which is what makes the mapping testable without a Filestore, an SttConfig, or a live client, per the repo's rule about extracting pure decision functions. AUDIO_NO_URL_NOTE moves to media.rs alongside the note it belongs to. Tests cover all three outcomes with and without an STT line, and assert that neither URL-less outcome emits a url: line. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NhMdtp4VVrqozRfx8uWxjz
Three contradictions between the published tables and the implementation, all introduced or exposed by this branch. The video table claimed Discord returns a presigned S3 URL when a filestore is configured. discord.rs always emits attachment.url for video and excludes video from the filestore branch outright, because the CDN link already resolves without credentials. The table now says so, and explains why. filestore.md still described video as never uploaded and gateway uploads as text-only. Slack video is presigned as of this branch, since neither Slack URL form is fetchable by the agent, and gateway audio uploads through the buffered single PUT. The behaviour table gains audio rows and splits video by platform. An operator choosing a storage configuration from these tables would otherwise provision for behaviour the application does not deliver. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NhMdtp4VVrqozRfx8uWxjz
The audio arm existed twice, once in run_gateway_adapter and once in process_gateway_event, and both reviewers noted the same thing: nothing exercises either call site. The copies had already drifted twice in this branch's own history, once as a duplicated read-failure block and once as an STT-failure warn! present on only one side, which is the failure mode the duplication produces rather than a coincidence. gateway_audio_blocks is now the only arm. Both entry points hand it the attachment, the byte result and their config, so there is no second copy left to drift. That includes the filestore upload, which is where the content-type defect earlier in this branch diverged. Two tests drive the real arm. STT disabled with no filestore reaches neither the network nor AWS, so the read-failure and passthrough cases run in CI rather than under the #[ignore] that the repo requires of tests touching either. Both cfg branches compile and both tests pass with and without the filestore feature. Still uncovered: the filestore-success and STT-enabled outcomes, which need a fake for an S3 client and an HTTP endpoint. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NhMdtp4VVrqozRfx8uWxjz
|
All findings from both review rounds are addressed at From the first review
Two notes on that round. The ordering point was worse than cosmetic. On the CI half of #3: From the second reviewF1, audio classification. Fixed in F2, storage documentation. Fixed in F3, per-message work budget. Declined for this PR, reasoning added to Accepted Residual Risks. The numbers hold: the loop awaits serially, F4, routing coverage. Addressed by removing what made it dangerous. The audio arm existed twice, and the copies had already drifted twice in this branch's own history: the duplicated read-failure block and an STT-failure Two tests drive that arm directly. STT disabled with no filestore reaches neither the network nor AWS, so they run in CI rather than under the Still uncovered, stated plainly: the filestore-success and STT-enabled outcomes, which need a fake for an S3 client and an HTTP endpoint. The MIME-fallback boundary is covered as pure tests on
|
|
Re-tested at F1 — audio classifier now falls back to filename extension (verified)Uploaded the same 18785-byte m4a payload as before, but this time named Prompt block delivered: Agent's reply: "content_type: I also ran the negative case ( The commit message's rationale for returning F2 — docs matrix now matches implementation (verified)
F3 — documented rather than code-changed
F4 — gateway paths now share one seam (verified via code review)
TC-1 re-check on new head — still passesVideo with filestore configured: Verdict — LGTM at
|
|
Important CHANGES REQUESTED What This PR DoesThis PR makes inbound audio available to the agent independently of speech-to-text and makes Slack video use a credential-free presigned URL when filestore is configured. It also centralizes attachment metadata formatting, MIME fallback, and gateway audio routing. How It Works
Findings
Finding Details🟡 F1: Preserve the configured-filestore failure reason
This makes an outage, bad credentials, presign failure, or size rejection look like a missing configuration problem and directs the agent/operator to the wrong remediation. Keep enough typed outcome information to distinguish 🟡 F2: Synchronize the remaining user-facing contractsThe PR updates the new inbound-attachment and STT pages, but other supported references still contradict the reviewed behavior: Feishu says audio is skipped when STT is disabled or fails; the Google Chat schema says STT-disabled audio becomes a transcription-failed note; and the config reference says a no-filestore build leaves all behavior unchanged. The new gateway path instead always emits an audio metadata block, with no STT line when STT is disabled. The filestore failure table also promises a degraded availability hint for a configured upload failure, while the current audio path produces the misleading NoStore configuration instruction described in F1. Update these references together with F1 so operators and platform consumers receive one accurate matrix. 🟢 F3: Good convergence of the earlier fixesThe current head uses Baseline Check
Addressing External Reviewer Feedback@dogzzdogzz
The reviewed head resolves the duplicate read-failure block, preserves gateway audio MIME, adds extension fallback, centralizes the two gateway arms, and updates the primary storage matrix. The per-message work-limit concern is documented by the author as a broader filestore concern and is not re-raised here. F1 is a separate configured-store failure path: it was introduced by collapsing upload outcomes to @antigenius0910
Those improvements remain present at the reviewed head. This round adds the remaining configured-filestore failure semantic and the references outside the pages updated by the follow-up. 5. Three Reasons We Might Not Need This PR
What's Good (🟢)
|
upload_bytes_and_presign returned None both when the audio exceeded the configured max_file_size and when the upload or presign failed, and gateway_audio_blocks mapped every None to AudioOutcome::NoStore. That outcome carries "configure a filestore to give the agent a downloadable link", so an outage, bad credentials, a presign failure or a size rejection all told the operator to configure a filestore that was already configured, and pointed the agent at the wrong recovery. The helper now returns Result<_, AudioStoreError> distinguishing TooLarge from UploadFailed, and audio_outcome() maps Option<&Result<..>> to the outcome: None still means no filestore, which is a different thing from one that refused. Each failure carries a note that says what actually happened. Keeping that mapping pure is what makes the configured-but-failed cases testable at all, since reaching them through the real path needs an S3 client. Discord and Slack are unaffected: their None falls back to the platform URL rather than to this note. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NhMdtp4VVrqozRfx8uWxjz
Four references outside the pages this branch had already updated still described superseded behaviour. feishu.md said audio is silently skipped when STT is disabled or fails, and the Google Chat platform schema said STT-disabled audio is forwarded as a transcription-failed note. Both predate the passthrough: the block is now always emitted, and the transcription-failed line appears only when STT is enabled and fails. config-reference.md said a build without the filestore feature leaves all behaviour unchanged. It leaves filestore behaviour unchanged, but inbound audio still produces a block either way, which is the distinction an operator reading that line needs. filestore.md's failure table promised that a configured store which fails always yields a hint saying the file exists but is unavailable. It now spells out the three gateway audio rows, including that a configured store which fails is not reported as an absent one. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NhMdtp4VVrqozRfx8uWxjz
`is_audio_mime` is public again at its original signature. It was `pub` at the merge base, `media` is a `pub mod`, and the crate carries no `publish = false`, so narrowing it to `pub(crate)` was a compile break for an external consumer even though the behaviour never changed. The doc now points adapters at `audio_mime`, which is the one that also reads the extension when a platform sends a missing or generic type. Nothing this branch newly added is public. The prechecks already separate a non-2xx response, a Content-Length overrun, and a reported-size overrun. Once `bytes_stream()` starts, two more failures are possible: the platform's body stream dies, and the bytes actually read overrun `max_file_size`, which is the only authoritative measurement. Both returned a bare `anyhow::Error` and both were flattened into `PresignError::UploadFailed`, so the agent read "the upload did not complete" when the truth was that the platform withheld the bytes or that the file was over the limit. That sends diagnosis to the wrong component, and it is worst on exactly the chunked or mis-sized files the measured-size handling exists for. `stream_upload_and_presign` is itself `pub`, so giving it a typed result would have repeated the visibility mistake above. A `StreamUploadCause` rides in the error chain instead, and `presign_error_for_upload` classifies with `downcast_ref`. The cause is the source rather than the outer context, so every `Display` string and every log line stays byte-identical. Four sites are tagged, including "stream produced no data", whose own comment already said it indicates a download failure. The classifier is a pure function so the mapping is tested apart from S3: one test drives all three causes through it, through `AudioStoreError`, and on to the note the agent reads. What it does not cover is named in the PR follow-ups rather than implied away: the one line that calls the classifier, and an integration test for a genuinely interrupted stream, which needs a filestore double the crate does not have. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NhMdtp4VVrqozRfx8uWxjz
`cap_presigned_ttl` raises a configured `0` to `1` second because S3 rejects `X-Amz-Expires=0` outright, and its unit test pins that. Two references still said the configured value is never raised, so an operator reading either one would conclude the opposite of what the code does. Both now name the exception and the reason for it. Normalising zero was deliberate rather than an oversight: a URL that cannot work at all is worse than one that expires immediately, which is why this is a documentation correction and not a behaviour change. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NhMdtp4VVrqozRfx8uWxjz
`run_gateway_adapter` built every attachment block inside the `msg = ws_rx.next()` arm of its select, with `tasks.spawn` only afterwards. The audio branch awaits `gateway_audio_blocks`, which awaits an upload when a filestore is configured, and the streaming path allows 600 seconds. So while a store was slow or unavailable the socket read nothing else. Two details made that worse than it looks. On main, with STT disabled, this arm was a single `debug!`, so for a filestore deployment with STT off it was a new blocking remote call rather than an inherited one. And the slash-command handling sat after the attachment loop, so a `/cancel` arriving behind a large upload waited on that upload too, which is exactly when someone sends one. The loop is now `assemble_attachment_blocks`, awaited inside the spawned per-event work; the receive arm keeps only the two clones that work needs. Slash commands therefore short-circuit before assembly, so an attachment riding on a `/cancel` is no longer uploaded and then discarded. The unified entry point was never affected, since `main.rs` already wraps each `process_gateway_event` in its own spawn, but it shares the helper anyway. The two inline copies had already drifted: one logged a rejected attachment, the other logged an unreadable text file, and neither logged both. The shared version logs both, which is the only behaviour difference on the WebSocket side. Naming the loop is also what made it testable, having had no test at all in either copy. One test drives a mixed list and pins that arrival order survives and that an attachment type with no branch contributes nothing rather than an empty block; the other pins a rejection with no filestore in play. What is still not covered, and is named in the PR follow-ups rather than implied away: a test proving a stalled upload cannot stall the loop. `Filestore` wraps a concrete `aws_sdk_s3::Client`, so nothing can inject an upload that never returns. The scheduling seam was the deliverable half. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NhMdtp4VVrqozRfx8uWxjz
|
Important CHANGES REQUESTED What This PR DoesThis PR makes audio attachments actionable independently of STT and routes Slack video through filestore when available. It also centralizes attachment formatting, MIME handling, failure notes, and gateway attachment assembly. How It WorksShared media helpers build sanitized audio/video blocks, retain measured stored sizes, and distinguish configured-store failures. Discord and Slack use URL-backed uploads; gateway adapters process held bytes and may upload them before dispatching the event. Findings
Finding Details🔴 F1: Preserve receipt order and reset boundaries
The same gap bypasses 🟡 F2: Bound pre-dispatch attachment workEvery accepted WebSocket event immediately starts reads, base64 decoding, STT, and potentially a 600-second filestore transfer before Dispatcher capacity applies. There is no admission semaphore/queue, and 🟡 F3: Require an actual extension separator
🟡 F4: Reattach the constant documentationThe first two doc-comment lines describe the 1000-turn guard but are now attached to 🟢 F5: Prior fixes remain effectiveThe reviewed head retains public Baseline Check
Addressing External Reviewer Feedback@dogzzdogzz
The current head centralizes gateway audio handling, preserves actual MIME, handles explicit non-audio MIME conflicts, and updates the attachment matrix. F1/F2 identify the new scheduling consequence of moving that work outside the receive loop; this is distinct from the earlier duplicate-arm issue. @antigenius0910
Those behaviors remain present. This round re-reviewed the later gateway scheduling refactor and does not dispute the earlier adapter-path validation. @howie
The current source retains the typed Reviewer Aggregation
5. Three Reasons We Might Not Need This PR
What's Good (🟢)
|
There was a problem hiding this comment.
Important
CHANGES REQUESTED
Consolidated review: #1460 (comment)
| #[cfg(feature = "filestore")] | ||
| let filestore = filestore.clone(); | ||
|
|
||
| tasks.spawn(async move { |
There was a problem hiding this comment.
🔴 F1 - Preserve receipt order and reset boundaries
This task awaits attachment assembly before it reaches Dispatcher::submit. A slow earlier event can therefore be submitted after a later event in the same thread, despite the Dispatcher FIFO contract. A /reset in between only cancels messages already submitted, so the earlier task can also enter the fresh post-reset session.
Requested change: assign an ingress sequence/generation before spawning, commit same-thread events in sequence, and cancel or invalidate pre-reset work. Add delayed-assembly ordering and reset-race tests.
| #[cfg(feature = "filestore")] | ||
| let filestore = filestore.clone(); | ||
|
|
||
| tasks.spawn(async move { |
There was a problem hiding this comment.
🟡 F2 - Bound attachment work before dispatch
Each received event starts attachment reads, decoding, STT, and possible filestore I/O before Dispatcher capacity applies. This JoinSet has no admission bound and is only drained on shutdown or reconnect, so a burst can create unbounded concurrent work and retain task results for the connection lifetime.
Requested change: add bounded admission/backpressure before assembly, continuously reap tasks, and define a safe overload policy with a regression test.
| /// `ogv`), so this never claims an attachment `is_video_file` should handle. | ||
| #[cfg_attr(not(any(feature = "slack", feature = "discord")), allow(dead_code))] | ||
| fn audio_mime_from_extension(filename: &str) -> Option<&'static str> { | ||
| match filename.rsplit('.').next()?.to_lowercase().as_str() { |
There was a problem hiding this comment.
🟡 F3 - Require an actual filename extension
rsplit('.').next() returns the whole filename when no dot exists. With absent or generic MIME, a dotless filename such as mp3 is therefore classified as audio/mpeg and can be routed through STT/audio passthrough even though it has no extension.
Requested change: use rsplit_once('.') and require both a non-empty stem and extension. Add dotless recognized-name regression tests.
|
|
||
| /// Hard cap on consecutive bot messages in a channel or thread. | ||
| /// Prevents runaway loops between multiple bots in "all" mode. | ||
| /// Named so a test can pin it: the agent fetches this link unaided, which is |
There was a problem hiding this comment.
🟡 F4 - Reattach the constant documentation
The existing first two doc-comment lines describe the consecutive-bot-turn guard, but these new lines extend that block so it now documents DISCORD_CDN_NOTE; MAX_CONSECUTIVE_BOT_TURNS has no documentation.
Requested change: keep only the CDN explanation above DISCORD_CDN_NOTE and move the turn-limit explanation directly above MAX_CONSECUTIVE_BOT_TURNS.
`rsplit('.').next()` returns the whole string when there is no dot, so a
file literally named `mp3` matched the extension fallback and was handed to
STT on its name alone. `rsplit_once` plus a non-empty stem makes the dot
required.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NhMdtp4VVrqozRfx8uWxjz
Hoisting DISCORD_CDN_NOTE to module scope inserted it between MAX_CONSECUTIVE_BOT_TURNS and the doc comment describing it, so the bot-turn cap read as documentation for the CDN note. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NhMdtp4VVrqozRfx8uWxjz
…assembly Moving attachment fetches into the spawned per-event task kept object storage off the WebSocket receive path, but the serial receive path was also what made three properties true for free, and all three broke with it: - same-thread events reached the dispatcher in arrival order, so a voice note taking 30s to upload could not be overtaken by the text sent after it - `/reset` only had to cancel buffered messages, because nothing else was in flight; a message still being assembled would now submit into the session created after the reset - exactly one attachment fetch ran at a time PreDispatchOrder restores the first two: a ticket is taken on the receive path, in arrival order, and carries the session generation it was taken in. A ticket waits for its predecessor before the dispatcher handoff (never before the fetch, which is the part meant to run concurrently) and drops itself if `/reset` has since bumped the generation. Dropping a ticket releases its successor, so a cancelled or panicking task cannot wedge a thread. Bounding is a semaphore of 4 concurrent fetches, JoinSet reaping on every event rather than only at shutdown, and load shedding past 32 pending events: the message still reaches the agent, carrying the same undelivered line a platform-side rejection produces, because shedding a user's text is worse than shedding the file attached to it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NhMdtp4VVrqozRfx8uWxjz
Resolved docs/inbound-attachments.md: the support matrix keeps this branch's "file + STT" wording and its Slack video row, and takes upstream's new LINE WORKS row, which is a gateway platform and so gets the same audio handling.
|
Important CHANGES REQUESTED What This PR DoesThis PR makes inbound audio actionable independently of STT and gives Slack video a credential-free presigned URL when filestore is configured. It also centralizes attachment formatting, media error handling, and gateway attachment assembly. How It WorksShared media helpers classify and sanitize attachment metadata, retain measured upload sizes, and render platform-specific fallback notes. The gateway moves attachment work out of the WebSocket receive loop, limits concurrent fetches, and uses Findings
Finding Details🟡 F1: Make the reset fence cover dispatcher enqueue and retry
The same generation bump leaves the old ticket tail attached. A post-reset event therefore waits for a stale attachment assembly to finish before it can reach the new session, potentially for the upload timeout. This contradicts the documented guarantee that 🟢 P1: Earlier attachment hardening remains effectiveThe current head keeps the successful earlier fixes: MIME fallback is bounded, prompt-visible metadata is sanitized, public Baseline Check
Addressing External Reviewer Feedback@dogzzdogzz
Addressed in the current head: gateway audio routing is shared, gateway uploads preserve MIME, missing/generic MIME fallback is bounded, ordering is paired per attachment, and the affected documentation was updated. The broader attachment-work budget remains a documented cross-type follow-up; it is not the reason for this verdict. @antigenius0910
Those adapter-path improvements remain present. The later receive-loop offload and ticket implementation require a separate review; this round identifies the remaining reset handoff race in that newer code. @howie
Verified as addressed in the current head: download failures retain a truthful note, stored attachments carry their measured byte count, and the turn-boundary ADR identifies transcript blocks rather than relying on the old block count. Reviewer Aggregation
5. Three Reasons We Might Not Need This PR
What's Good
|
Checking the ticket generation once before `submit` left two holes, both of them mine from the previous round. `submit` parks when the thread's queue is full. A `/reset` landing during that park drops the consumer, which turns the parked send into the `SendError` that `submit` transparently retries, and the retry creates a fresh consumer belonging to the session the reset just started. A message admitted before the reset therefore opened the session after it. The check now races the handoff instead of preceding it: the generation lives in a `watch` channel, so a ticket parked in the handoff is told about the reset rather than only being able to look before it starts waiting, and the handoff future is abandoned. That is safe because a parked `mpsc` send has enqueued nothing. The second hole is the tail: bumping the generation left the discarded events chained ahead of the ones that followed, so the first message of a new session waited out an upload from the old one, up to the streaming timeout. `reset` now clears the tail as well. Both are covered by tests that fail when the fix is reverted: dropping the reset branch from the handoff hangs the parked-reset test until its timeout, and leaving the tail attached fails the post-reset ordering test. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NhMdtp4VVrqozRfx8uWxjz
|
Important CHANGES REQUESTED What This PR DoesThis PR makes inbound audio actionable independently of STT and gives Slack video a credential-free URL when filestore is configured. It also moves gateway attachment assembly off the WebSocket receive loop, adding ordering tickets, bounded fetch concurrency, and load shedding. How It WorksShared media helpers sanitize metadata, retain measured stored sizes, and distinguish storage outcomes. On the gateway WebSocket path, each event takes a Findings
Finding Details🟡 F1: Cancel stale attachment preparation on reset
Reproduction: hold all four permits with slow pre-reset attachment uploads, send Requested change: make reset cancellation cover semaphore acquisition, attachment assembly, and pre-dispatch side effects, not only 🟡 F2: Preserve an admitted colocated attachment while it waitsThe semaphore is acquired before This is below the 32-event shedding threshold, so it contradicts the changed documentation saying that events queued for a fetch slot lose nothing. Requested change: establish a source lease before a task can wait behind remote work, for example by reading/pinning admitted colocated bytes with an explicit bounded-memory policy, or by extending the store lease through queued processing. Add a regression test where the queue delay exceeds the media TTL and assert that an admitted attachment is still available. Update the queue documentation to match the selected overload behavior. 🟢 F3: Earlier attachment hardening remains intactThe current source retains the previously requested public wrappers, bounded MIME fallback, measured upload sizes, typed storage failure notes, and prompt-structure sanitization. The ticket/handoff tests cover reset during a parked Dispatcher send; the missing coverage is reset during the earlier semaphore and assembly phase. Baseline Check
Addressing External Reviewer Feedback@dogzzdogzz
Those routing, MIME, content-type, and documentation fixes remain in the reviewed head. F1 and F2 are later scheduling consequences of moving gateway assembly out of the receive loop; they are distinct from the earlier duplicate-arm issue. @antigenius0910
Those adapter-path improvements remain present. This review evaluates the later current-head gateway scheduler and does not dispute that earlier live validation. @howie
The current source retains the corrected download-failure note, measured-size propagation, and transcript-based rollback documentation. The findings above concern the subsequent semaphore and reset implementation. 5. Three Reasons We Might Not Need This PR
What's Good
|
…eueing Two holes in the pre-dispatch scheduling, both reachable without a filestore. The reset fence covered the dispatcher handoff but nothing before it, so a discarded event went on holding a fetch slot, uploading bytes nobody would read, and could still create a forum topic. The first event of the new session then queued behind exactly that work. The fence now wraps the whole spawned body, so cancellation returns the slot and the source budget at once. One edge remains and is documented: a remote call already in flight may still take effect on the platform. Separately, the fetch slot was acquired before the colocated source was read, and the gateway store evicts media 120s after it lands. Four stalled uploads could therefore hold a fifth event past its source's lifetime, and the agent got a read failure for an attachment that existed when the event arrived, well under the shedding threshold. Sources are now read at admission, ahead of the queue, with an explicit 256 MiB budget bounding what admitted events hold; over that budget an event is admitted without its attachment bytes rather than queued. Both are covered by tests that fail when the fix is reverted: fencing only the handoff leaves a cancelled task holding the slot until the test times out, and reading inside assembly turns the surviving attachment into a read failure. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NhMdtp4VVrqozRfx8uWxjz
|
Important CHANGES REQUESTED What This PR DoesThis PR makes inbound audio actionable independently of STT and gives Slack video a credential-free presigned URL when filestore is configured. It also centralizes attachment rendering, MIME handling, failure notes, and gateway pre-dispatch scheduling. How It WorksShared media helpers build sanitized audio/video blocks, preserve measured stored sizes, and select platform-specific fallback notes. The gateway moves attachment assembly out of the WebSocket receive loop, preserves per-thread arrival order, cancels stale work on Findings
Finding Details🟡 F1: Charge the retained bytes, not an advisory size
Requested change: reserve a conservative upper bound before reading each event, or perform bounded reads that can charge actual bytes before retaining them. Reconcile the guard to actual retained bytes and shed/reject safely when the event cannot fit. Add a regression with under-reported attachment sizes proving the configured limit cannot be bypassed. 🟡 F2: Sanitize the rejection reason before it enters a prompt line
Requested change: apply the same single-line structural-character filtering to 🟢 F3: Full stale-work cancellationThe latest revision wraps the spawned preparation body in 🟢 F4: Consistent attachment outcomesThe shared decision function keeps the successful-upload measured size and distinguishes absent storage from configured-store failures. This preserves the earlier fixes for Discord download failures, Slack private URLs, and gateway no-store behavior. Baseline Check
Addressing External Reviewer Feedback@dogzzdogzzEarlier feedback on gateway-arm drift, MIME fallback, content type, documentation, and attachment work limits is retained in the current head: the audio arm is shared, explicit non-audio MIME conflicts stay out of STT, gateway uploads preserve MIME, and the attachment matrix was updated. The new F1 is narrower: the newly added byte limit is not a real memory bound when declared sizes are inaccurate. @antigenius0910Earlier live validation of generic-MIME audio routing and Slack video presigning remains useful for those adapter paths. This review covers later gateway scheduling and prompt-construction changes at the new exact head. @howieEarlier findings about truthful Discord download failures, measured stored sizes, and the voice-only rollback contract are preserved as fixed in the current head. F1 and F2 concern the final pre-dispatch and prompt-sanitization changes added afterward. 5. Three Reasons We Might Not Need This PR
What's Good
|
There was a problem hiding this comment.
Important
CHANGES REQUESTED
Consolidated review: #1460 (comment)
| while tasks.try_join_next().is_some() {} | ||
| let has_attachments = !event.content.attachments.is_empty(); | ||
| let event_bytes: u64 = | ||
| event.content.attachments.iter().map(|a| a.size).sum(); |
There was a problem hiding this comment.
🟡 F1 - Charge retained bytes, not advisory sizes
GwAttachment.size is documented as advisory, but this value is reserved before source paths are read or inline data is decoded. Under-reported sizes can therefore admit and retain far more data than the documented 256 MiB limit.
Requested change: reserve a conservative upper bound or use bounded reads and reconcile the guard to actual retained bytes; add a regression proving under-reported attachments cannot bypass the limit.
| let (safe_filename, safe_mime) = crate::media::sanitize_attachment_meta(filename, mime_type); | ||
| format!( | ||
| "[System: attachment \"{}\" ({}, {}) was not delivered — {}]", | ||
| safe_filename, safe_mime, size_str, reason |
There was a problem hiding this comment.
🟡 F2 - Sanitize the rejection reason before formatting it
Filename and MIME are sanitized, but reason is copied verbatim into a [System: ...] prompt line. Telegram can derive this reason from an untrusted filename extension, so Unicode separators or bidi controls can bypass the new structural-character hardening.
Requested change: filter and bound reason with the same single-line policy before formatting it, and add a user-derived U+2028/U+2029 and bidi regression test.
…e reason
The 256 MiB budget introduced last round was charged from the platform's
declared size, which the platform is free to under-report. An attachment
claiming size 0 reserved nothing and then read its file in full, so the
budget bounded a number rather than the memory it exists to bound.
Reservations are now taken against an upper bound derived from the source
itself (file metadata, or the length base64 can decode to), and the read is
capped at what was reserved, so overshooting stays impossible even when that
bound is wrong. An attachment the budget refuses is delivered as its own
`not delivered` line rather than a read failure, since nothing failed to
read.
The rejection reason interpolated into that line is attacker-controlled and
went in verbatim: Telegram builds it from the filename extension
(`unsupported format: {ext}`), so a crafted filename could carry line breaks
and bidi overrides into the prompt. The filename beside it was already
sanitized; that sanitizer is now a reusable fragment helper and the reason
goes through it too.
Both are covered by falsifiers: charging the declared size again lets a
second attachment reporting size 0 past the budget, and interpolating the
reason verbatim restructures the prompt line.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NhMdtp4VVrqozRfx8uWxjz
|
Important CHANGES REQUESTED What This PR DoesThis PR makes inbound audio actionable independently of STT and routes Slack video through filestore when configured, while adding gateway scheduling and attachment-memory safeguards. How It WorksShared media helpers render sanitized attachment blocks and preserve typed storage outcomes. The gateway reads attachment sources before queued work, reserves a 256 MiB source budget, assembles blocks asynchronously, and uses ordering/reset controls before Dispatcher handoff. Findings
Finding Details🟡 F1: Keep the memory bound valid through assembly and queueing
A near-limit image can therefore hold the reserved source, its uncharged clone, and its uncharged base64 payload at once; after assembly, the queued payload remains while the reservation is released. This contradicts the stated peak/retained-memory guarantee and permits substantially more than 256 MiB under attachment load. Requested change: consume/move each source into assembly instead of cloning it, and account for any encoded/queued Baseline Check
Addressing External Reviewer Feedback@dogzzdogzz
The current head retains the shared gateway audio path, bounded generic-MIME fallback, MIME-preserving upload path, and updated attachment documentation. This review found a separate accounting gap in the newly added source-budget implementation. @antigenius0910
Those adapter-path improvements remain relevant, but this review independently evaluates the later gateway memory/scheduling changes at the current head. @howie
The current implementation retains typed download-failure handling, measured stored-size propagation, and the transcript-based batching documentation. None resolves the source-buffer clone and post-assembly queue accounting described above. 5. Three Reasons We Might Not Need This PR
What's Good (🟢)
|
…ueueing The 256 MiB budget covered the source buffers and nothing built from them, so it stopped bounding memory at the point the memory actually grew. Assembly cloned every source before use, which put a second copy of each attachment outside the reservation. `assemble_attachment_blocks` now takes the sources by value and moves each one into its block, so that copy cannot exist: the signature no longer offers a borrow to clone from. The reservation was also sized for the source alone, while an image holds its source and the base64 encoding of it at the same time, and a text file holds its source and the code block wrapping it. Reservations now cover the source plus whatever the block built from it retains, so the peak is what gets charged. Audio and video are unchanged: they carry a URL and metadata whatever their source weighs. Finally, the reservation was released when assembly returned, while the blocks it produced went on to sit in the dispatcher queue. That queue is bounded by message count, not by bytes, so nothing bounded it in size. The guards are now held for the whole task, and a per-message cap bounds what one message may hand over: 24 MiB of inlined payload, past which the attachment is described rather than inlined. The two limits cover different lifetimes and the documentation says which is which. Falsifiers: charging the source alone lets an image through a budget that has no room for its encoded copy, and dropping the per-message cap inlines a second image that should have been described. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NhMdtp4VVrqozRfx8uWxjz
|
Important CHANGES REQUESTED What This PR DoesThis PR makes inbound audio actionable independently of speech-to-text and routes Slack video through filestore when configured. It also centralizes attachment metadata construction, MIME classification, failure reporting, and gateway pre-dispatch scheduling. How It Works
Findings
Finding Details🔴 F1: Bound admission, not only attachment fetching
Requested change: apply a bounded admission policy before spawning work and discard attachment bytes before the task captures the event when shedding. Add a test that holds a same-thread dispatch send and proves task count and retained attachment payloads stay bounded past the limit. 🟡 F2: Charge the audio upload copy
Requested change: either reserve the transient upload copy or refactor the upload ownership so the original allocation is reused. Cover concurrent max-size audio uploads and assert the claimed 256 MiB bound includes the peak. 🟡 F3: Count only payload that will be inlinedThe cap check runs before the text-file branch determines whether bytes exceed Requested change: decide the text delivery mode before charging the inline cap, charging only the branch that retains text in the outgoing block. Add a filestore-enabled mixed text-plus-image regression test. 🟡 F4: Keep the gateway video contract consistentThe same document first states that gateway video is rejected as unsupported, which agrees with the Requested change: correct the pre-dispatch text to describe the implemented gateway types, or implement and test the advertised video path. 🟢 F5: Prior ordering and reset fixes holdThe current WebSocket implementation takes tickets on receipt, uses a generation watch to cancel pre-reset work, detaches the old tail on reset, and limits active attachment fetches. The targeted ordering, reset-handoff, stale-work cancellation, and source-survival tests substantiate those earlier corrections. Baseline Check
Addressing External Reviewer Feedback@dogzzdogzz
The current head retains the shared gateway audio path, explicit-non-audio MIME handling, MIME-preserving upload, measured stored sizes, and updated attachment documentation. The remaining F1-F3 are new defects in the later scheduler and memory-accounting design, not a restatement of the earlier adapter-routing concerns. @antigenius0910
Accepted as useful validation for those adapter paths. It predates this exact head; the current findings concern later gateway pre-dispatch changes and do not dispute the observed Slack behavior. @howie
The current source preserves the earlier failure, size, and batching fixes. The final source-budget revision still leaves the task-admission, audio-copy, and externalized-text accounting gaps identified above. Reviewer Aggregation
5. Three Reasons We Might Not Need This PR
What's Good
|
There was a problem hiding this comment.
Important
CHANGES REQUESTED
Consolidated review: #1460 (comment)
| let mut guard = ticket.guard(); | ||
| let fetch_slots = fetch_slots.clone(); | ||
|
|
||
| tasks.spawn(async move { |
There was a problem hiding this comment.
🔴 F1 - Bound admission, not only attachment fetching
The 32-event check only chooses shed_attachment_blocks; this unconditional spawn still captures the attachment vector, including inline data, and queues behind the same-thread ticket when dispatch is backpressured. A burst can therefore grow task-held payloads without bound after shedding begins.
Requested change: enforce a bounded admission policy before spawning and discard attachment bytes before a shed task captures the event. Add a backpressured-dispatch regression test proving both task count and retained payloads remain bounded.
| #[cfg(feature = "filestore")] | ||
| let stored = match filestore { | ||
| Some(fs) => Some( | ||
| crate::media::upload_bytes_and_presign(filename, &bytes, Some(mime_type), fs).await, |
There was a problem hiding this comment.
🟡 F2 - Include the audio upload copy in the reservation
Audio reserves only its source bytes, but this borrowed call reaches ByteStream::from(data.to_vec()), creating a second full buffer while the original remains live for optional STT. Four concurrent 20 MiB uploads add 80 MiB outside the documented 256 MiB limit.
Requested change: reserve the transient copy or refactor ownership to avoid it, then cover concurrent max-size gateway audio uploads.
| // Charged before the payload is built, so the cap bounds the peak and not | ||
| // just what survives it. | ||
| if let Ok(ref bytes) = bytes_result { | ||
| let payload = inline_payload_bytes(&att.attachment_type, bytes.len() as u64); |
There was a problem hiding this comment.
🟡 F3 - Charge only text that is actually inlined
This check runs before the text branch decides whether a file above TEXT_INLINE_LIMIT will be externalized through filestore. The source bytes consume the 24 MiB inline allowance even though the queued block is only a URL, so a later image can be incorrectly replaced by a payload-limit notice.
Requested change: select the text delivery mode before charging the inline budget and add a filestore-enabled mixed text-plus-image regression test.
| dispatcher. The blocks themselves live on in the dispatcher's queue, which is | ||
| bounded by message count (`max_buffered_messages`, 10 per thread) and not by | ||
| size, so the per-message inline cap is what bounds it in bytes. Only the types | ||
| that inline bytes are charged against that cap: audio and video carry a URL and |
There was a problem hiding this comment.
🟡 F4 - Make the gateway video contract consistent
This section says gateway audio and video carry URL metadata, but the earlier Unsupported Types section and assemble_attachment_blocks both reject gateway video. The two contracts cannot both be true.
Requested change: document only the implemented gateway attachment types here, or add and test the advertised video forwarding path.
…holds Three accounting gaps, all reachable without a filestore except where noted. Shedding described an attachment from metadata but left its bytes in the event the task then captured, so an event that was shed precisely because the broker was over its limit went on holding the payload through an unbounded wait for its turn. The limit named memory and bounded none. The bytes are now released on the receive path, before any task can capture them. The inline cap charged every text file its full size before the text branch had decided whether a filestore would take it. A large text file delivered as a URL therefore spent an allowance the outgoing message never used, and a later image was refused for space that was free. The charge now follows the delivery decision, so only bytes that reach the prompt are counted. The reservation treated audio as source-only, but an upload copies its bytes into the request body while the original is still alive for STT. The same holds for text above the inline limit. Both now reserve for the copy. Documentation said gateway audio and video carry URL metadata. Gateway video is rejected before Core sees it and has no branch at all, so only audio is claimed now, and the two byte limits describe which bytes each one counts. Falsifiers: keeping the shed payload, dropping the upload charge, and charging externalized text against the inline cap each fail their own test. Not addressed, and called out in the PR body rather than implied: the number of spawned tasks is still unbounded. Bounding it means either dropping messages under load or one consumer per thread instead of one task per event, and both are decisions for a maintainer. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NhMdtp4VVrqozRfx8uWxjz
|
Important CHANGES REQUESTED What This PR DoesThis PR makes inbound audio available independently of STT and routes Slack video through filestore when configured. It also adds shared attachment rendering, typed storage outcomes, and gateway pre-dispatch scheduling and resource controls. How It WorksDiscord and Slack build audio blocks alongside optional transcripts. Gateway events read attachment sources before asynchronous assembly, then use ordering tickets, reset cancellation, a fetch semaphore, a source budget, and attachment shedding before dispatcher handoff. Findings
Finding Details🔴 F1: Account for the inline base64 representation
For a 20 MiB inline audio attachment with filestore, the reservation is 40 MiB for decoded source plus upload body, while the still-live base64 input adds about 26.7 MiB. Refused attachments retain the same input because the Requested change: include encoded input in peak accounting and consume or clear 🟡 F2: Bound admission rather than only attachment fetchingOnce Requested change: choose and implement a bounded admission policy before spawning work, or replace one-task-per-event with bounded per-thread consumers. Add a test that blocks dispatcher handoff and proves task count and retained event metadata remain bounded beyond the threshold. 🟢 F3: Useful progress on the prior memory defectsThe latest revision clears Baseline Check
Addressing External Reviewer Feedback@dogzzdogzz
The current head retains the shared gateway audio path, bounded MIME fallback, MIME-preserving uploads, and the updated attachment matrix. This round does not re-raise those resolved adapter-path concerns. @antigenius0910
Those adapter-path results remain relevant. The findings here concern later gateway scheduling and memory-limit changes, not the validated Slack routing. @howie
The latest change correctly discards shed attachment payloads and charges several transient copies. F1 shows that the inline base64 representation remains outside that accounting, and F2 remains open because task admission is explicitly unbounded. @ShinyChang
That acknowledgement is accurate but does not make the resulting resource exposure non-blocking. Please select and implement a bounded policy before merge. 5. Three Reasons We Might Not Need This PR
What's Good (🟢)
|
…itted An attachment can arrive as base64 on the event rather than as a colocated path, and that string is allocated when the event is parsed. Nothing charged it and nothing freed it, so it rode along through assembly and dispatcher handoff outside the budget that claimed to bound attachment memory. A refused attachment kept it too, which is the opposite of what refusing is for. The input is now taken off the event before it is decoded, on every path including the refusals, and the reservation covers it alongside the buffer decoded from it. Admission is now bounded as well. Until now the 32-event rule shed attachment bytes but still spawned a task per event, so a burst against a backpressured dispatcher grew task state and event text without limit. Past 256 events in preparation the broker refuses the event and tells the sender to send it again. That refusal is a behavior change and worth being plain about. Bounding admission means either refusing work or stalling the socket, and stalling the socket would take `/cancel` down with it, which is the failure this path was built to avoid. Refusing is visible to the user and recoverable by them; the alternative was preparation growing until the process died. There is no config key, as with the other limits here: reaching 256 events in flight is a load problem to report, not a number to raise. Falsifiers: not charging the encoded input, keeping it on the event, and admitting past the limit each fail their own test. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NhMdtp4VVrqozRfx8uWxjz
|
Important CHANGES REQUESTED What This PR DoesThis PR makes inbound audio available as attachment metadata whether or not speech-to-text is enabled. It also routes Slack video through filestore when available, so agents receive a fetchable presigned URL instead of an unusable private Slack URL. How It WorksShared media helpers classify audio, sanitize attachment metadata, preserve typed storage outcomes, and render measured upload sizes. Discord and Slack add audio blocks beside optional transcripts; gateway paths prepare attachment blocks, optionally upload gateway audio to filestore, and use WebSocket-side ordering, reset, and resource controls. Findings
Finding Details🟡 F1: Charge the rendered text, not only its input bytes
Requested change: calculate the queued text payload after conversion, or conservatively account for the maximum lossy expansion before admitting it. Add a malformed UTF-8 boundary test that proves the rendered block cannot exceed the inline limit. 🟡 F2: Reject a source that exceeds its reserved length
Requested change: detect a byte beyond the reservation and render a size/budget failure rather than publishing a corrupted partial attachment. Cover a file that grows after the metadata check. 🟡 F3: Give unified ingress the same shared admission boundaryThe unified bridge creates a task per incoming event, while Requested change: put the shared budget, admission counter, and fetch semaphore in 🟡 F4: Intercept unified control commands before attachment assemblyThe unified path calls Requested change: parse and handle control commands immediately after the event passes trust gating, before reading or uploading any attachment. Add a regression that verifies an attached Baseline Check
Addressing External Reviewer Feedback@dogzzdogzz
The current head retains the shared gateway audio helper, extension fallback limited to missing/generic MIME, real gateway audio content type, and updated attachment documentation. The current findings concern later resource-control code and the unified ingress path. @howie
The current source preserves typed @antigenius0910
Those observations remain useful for the Slack adapter paths. This review concerns the current gateway attachment lifecycle and resource bounds. Reviewer Aggregation
5. Three Reasons We Might Not Need This PR
What's Good
Validation
|
There was a problem hiding this comment.
Important
CHANGES REQUESTED
Consolidated review: #1460 (comment)
| // just what survives it. | ||
| if let Ok(ref bytes) = bytes_result { | ||
| let payload = | ||
| inline_payload_bytes(&att.attachment_type, bytes.len() as u64, has_filestore); |
There was a problem hiding this comment.
🟡 F1 - Account for rendered lossy text
This cap charges bytes.len(), but String::from_utf8_lossy below can turn each malformed byte into a three-byte replacement character. A 20 MiB invalid UTF-8 text file therefore passes the 24 MiB check and can queue about 60 MiB.
Requested change: charge the post-conversion text length (or a conservative maximum expansion) and add an invalid-UTF-8 boundary test.
| .await | ||
| .map_err(|e| e.to_string())?; | ||
| let mut bytes = Vec::new(); | ||
| file.take(limit) |
There was a problem hiding this comment.
🟡 F2 - Do not publish a capped prefix as a complete attachment
take(limit) stops successfully at the metadata length and never verifies EOF. If the colocated source grows or is replaced after source_upper_bound, this returns a truncated prefix that audio, text, or image handling treats as a successful file.
Requested change: probe for one additional byte and report a size/budget failure rather than processing a partial payload.
| _ => {} | ||
| } | ||
| } | ||
| let budget = SourceBudget::new(MAX_ADMITTED_SOURCE_BYTES); |
There was a problem hiding this comment.
🟡 F3 - Share unified ingress resource limits
This allocation is per process_gateway_event call, while the unified bridge spawns one call per incoming event. Unlike the WebSocket path, concurrent unified events therefore do not share the 256 MiB budget, event admission limit, or fetch semaphore.
Requested change: move these guards into shared GatewayEventContext state and apply admission before unbounded preparation tasks are spawned.
| let has_filestore = false; | ||
| let (sources, _guards) = | ||
| read_attachment_sources(&mut event.content.attachments, &budget, has_filestore).await; | ||
| let extra_blocks = assemble_attachment_blocks( |
There was a problem hiding this comment.
🟡 F4 - Handle unified control commands before attachment work
The unified route assembles blocks before the /reset, /cancel, and config-command checks below. An attached audio control command can therefore run new filestore and STT work that is discarded instead of short-circuiting like the WebSocket path.
Requested change: intercept control commands before reading or uploading attachments, and cover an attached /cancel or /reset regression.
…d text Four holes, two of them on the unified path, which had none of the controls the WebSocket path grew over the last several rounds. The unified bridge spawns a task per event and `process_gateway_event` built a 256 MiB budget inside each one, so every concurrent event could reserve the whole limit independently. A per-event budget is the same as no budget. The budget, the fetch semaphore and an admission counter now live on the shared event context, which is built once and cloned per event, so all three mean what they say. The same path also assembled attachments before it looked for `/reset`, `/cancel`, or a config command, so a command carrying audio could upload to object storage and run transcription, and then return without ever dispatching the result. Commands are now handled first, as they already were on the WebSocket path. A text file was charged its input size against the 24 MiB inline cap, but lossy UTF-8 conversion spends a three-byte replacement character on every malformed byte, so 20 MiB of invalid input rendered a 60 MiB block. Validity is now checked before the conversion, without allocating, and the charge follows what the text will render to. Anything charged before its bytes are read assumes the worst case, because that is the only safe assumption available at that point. Finally, reading a colocated source used `take(limit)`, which returns a prefix as a success. A file replaced between measuring its length and reading it would be delivered truncated, with no note that anything was lost. The read now looks one byte past the reservation and fails if it finds one. Falsifiers: charging malformed text as its input size, accepting the prefix, and giving each clone a fresh budget each fail their own test. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NhMdtp4VVrqozRfx8uWxjz
|
Caution This PR has been waiting on the author for more than 2 days (labeled @ShinyChang — You must add a new comment on this PR to remove the |
|
Still active. Here is where this stands. What changed in the last roundHead is now
Three of the four carry a test that goes red when the fix is reverted. The Gates on this head: full workspace suite passes except one pre-existing What is blockingThe PR carries One question on scopeMost of the recent rounds have been ingress resource controls rather than the |
What problem does this solve?
Discord Discussion URL: https://discord.com/channels/1491295327620169908/1491969620754567270/1529321810255151135
Two inbound attachment classes never reach the agent in a form it can act on. One
delivers nothing at all, the other delivers a link the agent cannot open. Both break the
same rule: an attachment block is only worth emitting if the agent can actually fetch
what it points at.
Audio delivers nothing. The audio branch is a dead end on all three inbound paths.
With STT enabled the agent gets a transcript and nothing else; with STT disabled it gets
nothing at all (Discord/Slack add a 🎤 reaction and drop the file, the gateway logs at
debug!and drops it). The original file never reaches the agent either way, so a skillcannot do anything a transcript cannot already express: no diarisation, no timeline, no
re-transcription with a domain glossary, no local ASR tuned for a non-English language.
Slack video delivers an unusable link.
slack.rsspecial-cases video inside theNotAnImagearm:slack_file_download_urlreturnsurl_private_download(orurl_private), both ofwhich require an
Authorization: Bearer <bot token>header that the agent does nothold. So the video branch steps around filestore to emit a URL that 403s, with no note
explaining why, while the sibling
elsebranch for PDF/ZIP does the right thing. Theagent cannot tell a dead link from a live one, so it fails silently. Discord is
unaffected (
cdn.discordapp.comneeds no credentials) and the gateway has no videobranch at all, so this is Slack-only.
Together these are the last two attachment classes without a working passthrough. Images
get an
[Image attachment]block with a URL, text files are inlined or uploaded, andsince #738 binary files (PDF/ZIP/docx) get a
[File: ...]block with a presigned URL.The concrete case: a user drops a meeting recording (
.m4a) into Slack and asks the agentfor minutes with speakers and action items. A transcript alone cannot produce that, and the
agent has no way to reach the file. Drop an
.mp4in the same thread and the agentreceives a link it will fetch and fail on.
Refs #738 (established the presigned-URL passthrough pattern this extends),
Refs #251 (established the current 🎤 reaction behaviour, which this keeps).
Review Contract
Goal
Every inbound attachment class hands the agent a location it can actually fetch. Two
classes fail that today: audio delivers nothing on any adapter, and Slack video delivers
a
url_private_downloadlink that 403s without a bot token the agent does not hold. Afterthis PR, an audio attachment always produces an
[Audio attachment]block regardless ofthe STT setting, and Slack video routes through filestore like every other non-image
attachment.
Non-goals
are unchanged. The two failure lines did change: the gateway's transcription-failure
string no longer interpolates the filename, because the adjacent block already carries it
sanitised, and the separate read-failure line is replaced by a
note:on the block.The transcript augments the file block; it never replaces it and is never replaced by it.
adding one here would make audio the only class that can be configured to silently
discard a user's file.
video branch.
Accepted Residual Risks
Each item below still needs an
Accepted by: <maintainer>line before merge. The authorcannot sign these and a reviewer cannot sign on a maintainer's behalf, so they are listed
unsigned on purpose rather than by omission.
Slack with no filestore still hands over a URL the agent cannot fetch. Mitigation:
the
note:line names the requiredAuthorization: Bearerheader verbatim, so the agentcan report why rather than discovering it as an unexplained 403 or an HTML login page.
This is strictly better than the current behaviour of dropping the file. Recovery:
configure a filestore, which makes it a presigned URL.
Slack video's
url:changes host on filestore deployments. A consumer pattern-matchingfiles.slack.comwould now see the filestore host. That consumer is matching on a URL itcannot currently fetch, so this surfaces an existing failure rather than creating one.
Recovery: revert, no migration.
One extra download per attachment on filestore + STT deployments. Bandwidth cost, not
a correctness risk; the file is fetched once for STT and once to stream into S3. Follow-up
named below.
The gateway audio path has no manual test. Discord and Slack were exercised end to end
(see Validation); no Telegram/Feishu/LINE/Google Chat deployment was available, so the
gateway arm rests on unit tests and on sharing the same builder as the two paths that were
tested.
The gateway's
[Voice message ...]strings no longer interpolate a filename at all.Originally listed here as a deliberate asymmetry with the sanitised blocks this PR adds,
and folded in during review instead of deferred, so the forging vector via a filename such
as
x\n[System]: ...m4ais closed. See Changes since the review below.No per-message audio work budget, and none added. A message may carry ten files on
both Slack and Discord,
max_file_size_mbdefaults to 250, and each streaming attempt hasa 600s timeout, so one message can in principle schedule multi-gigabyte serial transfer.
That is real, but it is not a risk class this PR creates:
download_and_upload_any_file,the PDF/ZIP/binary path already on
mainfrom feat: support non-image binary file attachments inbound (PDFs, office docs, video) on Slack/Discord #738, has no count or aggregate-byte budgeteither. Only text files are capped, at 5, and that cap exists to protect the prompt from
inlined content, which is why it is bypassed the moment a filestore takes over. Audio bytes
are never inlined, so the text cap's rationale does not transfer. Capping audio alone would
leave the same exposure one branch away and imply a bound the binary path does not honour,
so the budget belongs in its own change covering every filestore-bound attachment type.
Tracked separately rather than folded in here. Since the round at
da9890eathe gatewayWebSocket path does carry a bound, but a different one: at most four attachment fetches run
at once and attachment bytes are shed past 32 pending events. That bounds concurrency and
queue depth per connection, not bytes per message, so the exposure described above is
unchanged on Discord and Slack.
The two gateway pre-dispatch limits are not configurable. Four concurrent attachment
fetches and 32 pending events are compile-time constants. An operator who reaches the second
one sees attachment bytes stop being fetched with no knob to turn. Deliberate: they are safety
valves against a slow store, and a config key would invite raising them back toward the
unbounded behaviour they exist to prevent. Documented in
docs/inbound-attachments.md.Recovery: the limits release themselves as the queue drains, and nothing is persisted.
Acceptance Criteria
[Audio attachment]block on Discord, Slack, and thegateway with STT both enabled and disabled. With STT enabled the transcript rides on
top of that block when transcription succeeds; when it fails, Discord and Slack emit
the block alone and the gateway adds
[Voice message - transcription failed].url:line is a presigned URL that resolves withoutcredentials; with none, it is the platform URL plus a
note:naming its requirement,or absent entirely on the gateway.
url:resolves without credentials when a filestore isconfigured.
[Video attachment]output is byte-identical tomainfor every filenameand content type
mainitself would emit. The shared builder diverges in five ways,each one deliberate: it strips control and separator characters from the filename,
truncates the filename at 200 chars, renders a filename made entirely of stripped
characters as
unnamed, restricts the MIME to[A-Za-z0-9/-+.;= ]and truncates itat 100 chars, and renders an empty MIME as
unknown. A crafted filename or an exoticcontent type therefore yields different, safer output.
Pinned by
the_video_block_matches_main_except_where_sanitizing_makes_it_differ,which compares against a
main_video_blockreference reproducingmain's formatstring, plus
the_video_block_truncates_a_name_at_200_and_a_mime_at_100andthe_video_block_substitutes_unknown_for_an_empty_mime_and_strips_quoting.#738binary-file path ([File: ...]) is byte-identical tomain, including itsthree degraded hint strings.
[Audio attachment]block, on Discord, Slack, and the gateway.dispatcher in that order even when the voice note's upload is slow, and a
/resetsentwhile one is still being prepared drops it instead of letting it open the new session.
Pinned by
same_thread_events_reach_the_dispatcher_in_arrival_orderanda_message_being_prepared_when_reset_arrives_is_dropped.cargo clippy -p openab-core -- -D warningsis clean with and without thefilestorefeature, and
cargo testis green in both configurations.Follow-ups
download_and_transcribeownsits own download, while
stt::transcribealready accepts bytes, as the gateway path shows.Downloading once and feeding both removes the extra fetch and makes the two adapters match
the gateway's shape. Deferred because it is a change to the STT path, which this PR
deliberately does not touch.
[Video attachment]handling at all;whether it should upload video bytes to filestore the way it now does for audio is a
separate design question.
[File: ...]binary path still says nothing onTooLargeorDownloadFailed.Audio now explains both conditions in a
note:, so a 300 MB.m4aand a 300 MB.zipinone message get an explanation and total silence respectively. Closing it means changing
the
#738path, which AC-5 deliberately holds byte-identical, so it belongs in its ownchange.
.filter(|c| !c.is_control())call sites. They cover C0/C1 only, sothe separators and bidi formatters
splits_a_prompt_linehandles survive them. Four feedthe
#738blocks that AC-5 pins byte-identical; converting them is a behaviour change tothat path and is scoped out for the same reason.
store.rs's size ceiling disagrees with the STT ceiling. The two constants differ (20vs 25 MB) with no comment explaining which one an operator is actually bounded by. Not
touched here because neither value changes in this PR.
attachment_url_note_sizeand giving each adapter a row test closes the mutation thatrewrites the Slack fallback arm, and makes both note constants assertable. Three of the four
mutations the review listed survive regardless, because they delete or rewrite statements
inside
EventHandler::messageandhandle_messagethemselves: dropping theextra_blocks.extend(...)call, passingNoneinstead ofSome(bot_token)to the Slackupload, and swapping
stt_line = Some(...)forextra_blocks.insert(0, ...). Reaching thoseneeds a fixture that drives a whole handler with a fake platform client, which does not exist
in this crate today and is a larger change than this PR.
drives
StreamUploadCausethrough the classifier,AudioStoreError, and the note text forall three causes, which is the mutation that matters. It does not cover the one line inside
download_and_presign_any_filethat calls the classifier, and an integration test proving agenuinely interrupted body stream produces
SourceReadneeds a filestore test double:Filestorewraps a concreteaws_sdk_s3::Client, so nothing can inject a failing uploadtoday. That double is its own change.
main.rsalready wraps eachprocess_gateway_eventin its owntokio::spawn, so same-thread ordering and the/resetboundary there rest on tokio's scheduling exactly as they did on
main. Giving that path thesame ticket is a change to code this PR does not otherwise touch, and the gap is pre-existing
rather than introduced here.
[Image attachment]block now sanitises but has no test of its own. It isbuilt inline in the message handler rather than by a pure builder, so the coverage it has
is
sanitize_attachment_meta's own unit tests. Extracting it the way audio and video wereextracted is the follow-up.
Changes since the review at
cd38142Behaviour change: Discord and Slack audio block order.
media::audio_attachment_blocksnow owns the order of one attachment's blocks and all three adapters extend from it.
Discord and Slack previously inserted the transcript at index 0 of
extra_blocksfrominside the loop over attachments, so three voice notes rendered as
[t3, t2, t1, m1, m2, m3]: transcripts reversed and detached from the files they describe.Order is now transcript then metadata, per attachment.
message, or audio alongside another attachment type. A single audio attachment on its own
is byte-identical to before, which is why this order was chosen over metadata-first.
Behaviour change: audio classification.
media::audio_mimefalls back to the filenameextension when the platform omits the MIME, so a
clip.oggormeeting.m4aarriving with anempty or
application/octet-streamtype now enters the audio branch. It previously fellthrough to the image path and was dropped, or became a generic
[File: ...]block.Anything already labelled
audio/*is unchanged. STT now runs on these attachments whereit did not before, so a deployment with STT enabled will see transcripts appear for files
that were previously silent.
stt::transcribebuilds its multipart body withPart::mime_str(...).ok()?, which discards the request when the value does not parse.Admitting an attachment whose MIME is
""would have traded "audio silently dropped" for"audio always fails to transcribe".
Fix: gateway audio content type.
Filestore::upload_and_presignhardcodedtext/plain; charset=utf-8, so the same.m4awas stored asaudio/mp4from Slack andtext/plainfrom the gateway platforms. An earlier revision fixed this by adding a requiredcontent_typeargument, which was a compile break for anyone outside this crate, sinceopenab-corecarries nopublish = falseandpub mod filestoreis exported. Thetwo-argument method is restored with its original
text/plain; charset=utf-8behaviour, andthe new
upload_and_presign_with_content_typemirrors the argumentstream_upload_and_presignalready took. Everything this branch newly added for the adapters is
pub(crate). Onepre-existing
pubitem was narrowed alongside them, which was the same mistake inminiature; that is reverted below.
Fix:
presigned_ttlrendering. Every site rendered the lifetime asttl / 60, soanything under 60s reported "expires in 0 minutes". An earlier revision fixed this with a
60-second floor, which silently lengthened a configured authorization lifetime and had no
business riding along with attachment passthrough. The configured value is now passed through
untouched and the shared formatter reports seconds below a minute. The pre-existing 7-day cap
is unchanged, and at 60s and above the formatter emits exactly the string
ttl / 60produced,so the
#738hint stays byte-identical at every TTL that path could already render.Closed from Follow-ups. The four unsanitised
[Voice message ...]filenameinterpolations are gone. Two were the redundant read-failure line, dropped; the other two
no longer name the file, because the adjacent metadata block already carries it sanitised.
Changes since the review at
a44e8eaFix: Discord no longer erases a
DownloadFailed. When a configured filestore failedbecause the bot's own fetch of the CDN link failed, Discord fell back to that same link
carrying its ordinary "expires ~24h" note, telling the agent to fetch a URL the bot had just
proved it could not fetch.
DownloadFailedis now the one store outcome that overrides theplatform's "the agent can fetch this" claim.
TooLargeandUploadFailedstill fall backquietly, because the CDN link genuinely works in those cases.
Fix: the measured byte count wins over the platform's advisory size.
size_bytes:wastaken from the platform's reported size even when the file had just been streamed through the
filestore and counted on the way. Discord's
sizeis advisory and Slack's can be absent, sodownload_and_presign_attachmentnow returns aStoredAttachmentcarrying the count itmeasured, and the block renders that.
New: one decision function for the url / note / size row.
media::attachment_url_note_sizetakes the store outcome plus a
PlatformUrlsaying whether the agent can fetch the platform'sown link unaided, and returns the three values a block renders with. Both adapters route
through it, and each carries a test walking every row (
Ok,TooLarge,UploadFailed,DownloadFailed, no filestore) with no S3 client involved.DISCORD_CDN_NOTEandSLACK_URL_REQUIREMENTmoved to module scope so those tests can pin them literally; beforethe move, emptying either constant left the whole suite green.
Fix:
audio_mimecompared a half-normalised MIME.Audio/OGGread as an explicitnon-audio type and suppressed the extension fallback, so a correctly typed file was dropped
by its capitalisation. The value is lower-cased once, up front.
Hardening: the sibling blocks that still interpolated raw metadata. Both gateway
[System: attachment "..." was not delivered ...]sites now share oneundelivered_attachment_linebuilder that sanitises throughsanitize_attachment_meta, andDiscord's
[Image attachment]block sanitises the same way. The gateway line was the sharpercase, since a filename carrying a newline could forge a second
[System: ...]line. Slackalso now skips a file with no private URL instead of emitting a block with an empty
url:.mainand passed the filename and MIMEthrough raw, so this is a behaviour change outside the audio and video blocks the rest of
the PR adds. A filename or content type containing control or separator characters now
renders stripped, a filename longer than 200 chars renders truncated, one made entirely of
stripped characters renders as
unnamed, and a MIME outside[A-Za-z0-9/-+.;= ]rendersfiltered and truncated at 100. Ordinary filenames are byte-identical to before.
blocks is that attachment metadata reaches the prompt attacker-controlled. That argument
does not stop at the two blocks this PR happens to add, and the gateway line is the sharper
case of the two.
Tests: the paths the review showed were unpinned. An
stt_on_unreachable()fixture pointsSTT at a closed port, so the gateway's STT-failure arm runs deterministically with no network
and no mock; restoring the filename to that failure string now fails a test. The video block
gained a parity table against a
main_video_blockreference that reproducesmain's formatstring, so every divergence is declared rather than assumed, plus the 200-char and 100-char
truncation boundaries. A MIME-forging test pins what the allow-list actually protects: an
attacker-supplied
content_typecannot add a line or a secondurl:, though it does surviveas inert text.
Docs: the turn-boundary ADR's voice-only scenario. Adding a metadata block changed the
block count the ADR's rollback hatch keys on: with STT a voice-only arrival is two blocks so
extra_blocks.len() == 1 && prompt.is_empty()never fires, and without STT the single blockis the metadata, which the hatch would promote into the prompt slot. Section 3.1, the Scenario
D worked example (now shown for both STT states), and the hatch's two other citations now
identify the transcript block explicitly. ADR bumped to 0.7 with a changelog entry.
Docs: drift outside the diff.
telegram.tomlandline.tomlstill described core's onlyaudio action as transcription, while their four siblings were updated in this PR;
line.toml'svoice_sttalso contradictedline.md.filestore.md's size-refusal rowclaimed a note Discord never emits, and listed two Future Directions that ship here.
slack.md'sfiles:readrow omitted video and binaries.discord.tomlclaimed non-imagefiles are warned to the user, when the warning fires only for images that failed to download.
Two code comments were wrong in the same way:
audio_attachment_block's said a gatewayurlis always
None, andgateway_audio_blocks's misnamed the drift that motivated it (bothcopies emitted the read-failure block; what one copy dropped was the logging).
Changes since the review at
471cd64bThree of the four findings from that round are fixed here. The fourth is a scheduling decision
and is stated as open below rather than answered quietly.
Fix: the public MIME helper is public again.
is_audio_mimewaspubat the merge base,mediais apub mod, and the crate carries nopublish = false, so narrowing it topub(crate)was a compile break for an external consumer even though its behaviour neverchanged. It is public again at its original signature, with the doc pointing adapters at
audio_mime, which is the one that also reads the extension when a platform sends a missing orgeneric type. Worth being explicit that this reverses a narrowing an earlier round asked for:
that request and the compatibility request from the round before it were in conflict, and this
resolves it on the compatibility side. Nothing this branch newly added is public.
Fix: a post-response failure names the component actually at fault. The prechecks correctly
separate a non-2xx response, a Content-Length overrun, and a reported-size overrun. After
bytes_stream()starts, though, two more things can go wrong: the platform's body stream canfail, and the bytes actually read can overrun
max_file_size, which is the only authoritativemeasurement. Both returned a bare
anyhow::Errorand both were flattened intoPresignError::UploadFailed, so the agent readthe upload did not completewhen the truth waseither that the platform withheld the bytes or that the file was over the limit. That points
diagnosis at the wrong component, and it is worst on exactly the chunked or mis-sized files the
measured-size handling exists for.
stream_upload_and_presignis itselfpub, so changing its return type would have repeated thefirst finding. Instead a
StreamUploadCausenow rides in the error chain andpresign_error_for_uploadclassifies withdowncast_ref. The cause is the source rather thanthe outer context, so every
Displaystring and every log line is byte-identical to before.Four sites are tagged, including
stream produced no data, whose own comment already said itindicates a download failure. A test drives all three causes through the classifier and on to
the note the agent reads.
mid-transfer or under-reports a size. They previously got a note naming the store; they now
get one naming the platform or the size cap. No successful path changes.
Fix: the
presigned_ttldocs state the zero-second exception.cap_presigned_ttlraises aconfigured
0to1because S3 rejectsX-Amz-Expires=0outright, and its unit test pinsthat. Two references still said the configured value is never raised. Both now name the
exception and why it exists. This is a doc correction, not a behaviour change: normalising zero
was deliberate, since a URL that cannot work at all is worse than one that expires immediately.
Fix: object-storage transfer no longer runs inside the gateway receive loop. In
run_gateway_adapterthe attachment loop sat inside themsg = ws_rx.next()arm of the select,with
tasks.spawnafter it, so whilegateway_audio_blocksawaited an upload the socket readnothing else. Two details made it worse than it first looked. On
main, with STT disabled, thatarm was a single
debug!, so for a filestore deployment with STT off this was a new blockingremote call rather than an inherited one. And the slash-command handling sat after the loop, so a
/cancelarriving behind a large upload waited on that upload too, which is precisely when auser would send one.
The loop is now a named
assemble_attachment_blocksawaited inside the spawned per-event work.The receive arm keeps only the two clones that work needs. The unified entry point was never
affected, because
main.rsalready wraps eachprocess_gateway_eventin its owntokio::spawn,but it shares the helper anyway: the two inline copies had already drifted, one logging a
rejected attachment and the other logging an unreadable text file, with neither logging both.
The shared version logs both.
store no longer stalls receipt of later events. Slash commands now short-circuit before
attachment assembly, so an attachment riding on a
/cancelis no longer uploaded and thendiscarded. There is no reaction to re-time on this path; the 🎤 reaction is Discord's, not the
gateway's. One log line is new on the WebSocket path, the
text_fileread failure that onlythe unified copy used to report.
Tests the extraction made possible. The loop was inline in two entry points and had no test
at all; naming it is what allowed two. One drives a mixed list (a rejected attachment, an audio
attachment, an unrenderable sticker) and pins that arrival order survives and that a type with no
branch contributes nothing rather than an empty block. The other pins that a rejected attachment
is reported with no filestore in play at all.
On the regression test the finding asked for, proving that a stalled upload cannot stall the
loop: that one is still not deliverable, for the reason in Follow-ups.
Filestorewraps aconcrete
aws_sdk_s3::Client, so nothing can inject an upload that never returns. The schedulingseam is the part that was deliverable, and it is now explicit rather than implied.
Changes since the review at
da9890eaAll four findings from that round are fixed. Two of them are this branch's own doing, one round
old, and are described that way below rather than as neutral improvements.
Fix: receipt order and reset boundaries survive delayed assembly. Moving attachment fetches
into the spawned per-event task, the previous round's fix, did keep object storage off the
receive path. It also removed what made three properties true for free, because serial execution
on that path was itself the mechanism:
submitnow sits after an awaitthat can run for the streaming timeout, so a voice note could be overtaken by the text sent
after it, and the dispatcher's per-thread queue would take them in that order.
dispatch.rsstates no merging, splitting, or reordering as a broker invariant, and the batching ADR's
prohibited-transformations list names ordering inversion among the things the broker is not
authorised to do.
/resetonly had to cancel buffered messages, because nothing else was in flight. A messagestill being assembled would submit into the session created after the reset.
PreDispatchOrderrestores the first two. A ticket is taken on the receive path, in arrivalorder, carrying the session generation it was taken in. It waits for its predecessor immediately
before the dispatcher handoff, never before the fetch, which is the part meant to run
concurrently, and then drops itself if
/resethas bumped the generation since. Dropping aticket releases its successor, so a cancelled or panicking task cannot wedge a thread, and idle
threads are swept once more than 256 are tracked.
Bounding is three things: a semaphore of four concurrent fetches,
try_join_nextreaping onevery event rather than only at shutdown and reconnect, and load shedding past 32 pending events.
Shedding drops the bytes, not the message: the agent still gets the event, carrying the same
[System: attachment ...]line a platform-side rejection produces with the limit named as thereason. Losing a user's text to load shedding is worse than losing the file attached to it.
Both limits are compile-time constants with no config key, documented in
docs/inbound-attachments.mdas safety valves rather than tuning knobs, and the ADR's orderingclause now names the ticket instead of leaving serial execution implied.
were before the previous round's commit. New under load only: at 32 pending events attachment
bytes stop being fetched, which no deployment reaches without a store that is already failing
slowly. A message dropped by
/resetwhile being prepared is logged atinfo!and is notcounted in the reset reply's
Dropped n buffered message(s), which still counts bufferedmessages only.
Tests.
same_thread_events_reach_the_dispatcher_in_arrival_orderdrives the actual failure:the first event blocks on a stand-in for the fetch, the second runs, and the test asserts nothing
has reached the dispatcher before the first is released. Neutering
wait_for_turnfails it.a_message_being_prepared_when_reset_arrives_is_droppedpins the generation check in bothdirections, including that another thread's reset is not this thread's business.
a_dropped_event_releases_the_next_onepins the no-wedge property,a_second_thread_is_not_held_behind_the_firstthat the order is per thread, andan_idle_thread_stops_being_trackedthe sweep. The shed decision and its block are pinned byattachment_work_is_shed_only_once_the_queue_is_fullanda_shed_attachment_still_tells_the_agent_what_arrived, andthe_order_key_matches_the_one_reset_scopes_topins the one string the two halves have to agreeon.
The regression test for the original stall, that a hung upload cannot stall the receive loop, is
still not deliverable for the reason in Follow-ups:
Filestorewraps a concreteaws_sdk_s3::Client.Fix: a real extension is required before classifying audio by name.
rsplit('.').next()returns the whole string when there is no dot, so a file literally named
mp3matched theextension fallback and was handed to STT on its name alone.
rsplit_onceplus a non-empty stemmakes the dot required. A test covers
mp3,wav,ogg,.mp3, and.alongside the positivecases. Three other
rsplit('.')call sites inmedia.rshave the same shape; all three arepre-existing on
mainand none is on this branch's path, so they are left alone.It previously reached STT and the audio block; it now falls through to normal type handling.
Base moved:
upstream/mainmerged in. The LINE WORKS adapter (#1456) landed while thisround was in progress and conflicted in
docs/inbound-attachments.md, which is why the diff nowshows a row for a platform this PR never touched. The support matrix keeps this branch's
file + STTwording and its Slack video row, and takes upstream's new LINE WORKS row with thesame wording applied: LINE WORKS forwards audio as an
audioattachment through the gateway, soit goes through exactly the arm this PR changed and gets the metadata block whether or not STT is
on.
LINE WORKSis also added to the audio URL table's gateway enumeration, so the two tablesagree. Everything else in that commit merged cleanly; the only source overlap was one line in
gateway.rsaddinglineworkstoNON_EDITABLE_PLATFORMS.Fix: the turn-limit doc is back above its own constant. Hoisting
DISCORD_CDN_NOTEto modulescope two rounds ago inserted it between
MAX_CONSECUTIVE_BOT_TURNSand the doc commentdescribing it, so the bot-turn cap read as documentation for the CDN note. Comment move only.
Changes since the review at
8c4b0769One finding, and it is the other half of the round before it. The ticket introduced then fenced
/resetagainst work still being prepared, but the fence was a single check taken immediatelybefore the dispatcher handoff, which left two holes.
Fix: the fence covers the handoff instead of preceding it.
Dispatcher::submitparks when thethread's queue is full. A
/resetlanding during that park drops the consumer, so the parked sendreturns
SendError, andsubmittransparently retries it onto a consumer it creates fresh, whichbelongs to the session the reset just started. A message admitted before the reset could therefore
be the one that opened the session after it, which is the opposite of what the reset was asked to
do. The generation now lives in a
watchchannel, so a ticket parked in the handoff is told aboutthe reset rather than only being able to look before it starts waiting, and the handoff future is
abandoned when that happens. Abandoning is safe: a parked
mpscsend has enqueued nothing, and asend that already completed lands on the pre-reset consumer, which the reset aborts anyway.
Fix: a reset detaches the events that follow it. Bumping the generation left the discarded
events chained ahead of later ones, so the first message of a new session waited for a discarded
upload to finish before it could reach the dispatcher, up to the streaming timeout.
resetnowclears the thread's tail as well, so post-reset events start a fresh chain.
/resetcan race an attachment upload. Bothholes need a reset concurrent with in-flight preparation, so neither is reachable without one.
No successful path changes, and no config, schema, or wire-format change is involved.
Tests.
a_reset_during_a_parked_handoff_abandons_the_messagedrives the retry race with ahandoff that never resolves, standing in for the parked send, and asserts the reset releases it;
removing the reset branch from the handoff hangs it until the timeout.
a_post_reset_event_does_not_wait_for_pre_reset_workholds a pre-reset ticket open and asserts thepost-reset event has no predecessor and does not wait; leaving the tail attached fails it.
a_reset_before_the_handoff_abandons_the_messageanda_handoff_that_lands_first_counts_as_submittedpin the two non-racing outcomes.
The documented
/resetguarantee indocs/inbound-attachments.mdis updated in the same push tostate what the fence now actually covers, rather than the weaker property it described before.
Changes since the review at
487e59d8Two findings, both in the gateway pre-dispatch scheduling, both fixed.
Fix: a reset cancels the whole of a discarded event's preparation. The fence added in the
previous round covered the dispatcher handoff and nothing before it, so an event the reset had
already invalidated went on holding a fetch slot, uploading bytes nobody would read, and could
still reach
create_threadand create a forum topic. The first event of the new session thenqueued behind exactly that work at
fetch_slots.acquire(), which is the opposite of the guaranteethis PR's documentation claimed. The fence now wraps the whole spawned body, so cancelling returns
the fetch slot and the source budget at once and stops the side effects before they happen.
One edge remains rather than being claimed away: a remote call already in flight when the reset
lands, a forum-topic creation whose request has left the process, may still take effect on the
platform. Cancellation stops the broker from acting on the result, not the platform from having
received the request. That is stated in
docs/inbound-attachments.md.Fix: an admitted attachment is read before its event queues. The fetch slot was acquired before
the colocated source was read, and the gateway store evicts media 120 seconds after it lands,
sweeping every 30. Four stalled uploads could therefore hold a fifth event past its source's
lifetime, and when a slot finally opened the read failed, so the agent got a read-failure block for
an attachment that existed when the event arrived. This sat well under the 32-event shedding
threshold, so nothing announced it. Sources are now read at admission, ahead of the queue.
Holding those bytes is what the new admitted-source budget bounds: 256 MiB across admitted events,
measured at this point against the platform's declared sizes, which the next round replaces with the
bytes actually held. Over that budget an event is
admitted without its attachment bytes, taking the same undelivered line the pending-event limit
produces, rather than being queued. Reading before queueing would otherwise trade a read failure
for unbounded memory, which is not a trade worth making silently.
in-flight preparation. The source read is unconditional but strictly earlier than before, so the
only new cost is the memory the budget bounds. No config, schema, or wire-format change.
Tests.
a_reset_releases_the_fetch_slot_the_new_session_needsreproduces the reviewer's case:a stale task holds the only slot, the reset lands, and the test asserts the slot comes back;
fencing only the handoff leaves it held until the test times out.
a_reset_stops_the_work_before_any_of_it_runsasserts the body never starts, which is what makesthe forum-topic side effect unreachable.
an_admitted_attachment_survives_a_source_that_expires_while_it_queuesreads the source, deletes the file, asserts the deletion really took (or the test would prove
nothing), and then asserts the assembled block is the attachment rather than a read failure;
reading inside assembly again turns it into a read failure.
the_source_budget_bounds_what_admitted_events_may_holdand
an_abandoned_task_returns_its_source_budgetpin the budget and its release on cancellation.A note on the shape of this PR. This is the fourth consecutive round spent on gateway
pre-dispatch scheduling, all of it downstream of one decision: moving attachment assembly off the
WebSocket receive loop. Each round has found a real defect in that machinery, and the machinery now
runs to a ticket, a generation watch, a cancellation fence, a fetch semaphore, a byte budget, and a
shedding policy, none of which the audio and video contracts this PR exists for actually need. The
alternative that was available at the time, and still is, is to put assembly back on the receive
path and bound the stall with a much shorter timeout: that makes ordering, reset, and boundedness
correct by construction, at the cost of leaving the receive-loop finding only partly addressed. It
is worth a maintainer's explicit call whether this scheduling work should stay here or land as its
own change, rather than continuing to accrete inside an attachment-contract PR.
Changes since the review at
9c83db99Two findings, both mine, both introduced by the previous round's budget work. Both fixed.
Fix: the source budget is charged for bytes actually held. The 256 MiB budget reserved against
attachment.size, the size the platform declares in the event. A gateway platform is free tounder-report it, and an attachment declaring
size: 0reserved nothing and then read its file infull, so the budget bounded a number rather than the memory it exists to bound. Enough of them and
the limit is decorative. Reservations are now taken against an upper bound derived from the source
itself:
fs::metadatafor a colocated path, or the length base64 can decode to for inline data. Theread is then capped at what was reserved (
AsyncReadExt::take), so even an upper bound that iswrong cannot overshoot the reservation.
An attachment the budget refuses now renders as its own
[System: attachment ... was not delivered ...]line, naming the budget, instead of a read failure. Nothing failed to read, and telling theagent otherwise sends it to retry a fetch that was never attempted.
Fix: the rejection reason cannot restructure the prompt line.
undelivered_attachment_linesanitized the filename and the MIME type beside it, and then interpolated the reason verbatim. The
reason is attacker-controlled on the same footing:
telegram.rsbuilds it asunsupported format: {ext}, straight from the filename extension, so a crafted filename could putline breaks or bidi overrides into a
[System: ...]line the agent reads as broker-authored. Thefilename sanitizer is now a reusable fragment helper (
sanitize_prompt_fragment) and everyuntrusted piece of that line, reason included, goes through it.
now refuses per attachment rather than per event, so a message whose second attachment is refused
still delivers its first. Rollback: revert the commit.
Tests.
an_under_reported_size_cannot_bypass_the_source_budgetsends two attachments declaringsize: 0whose real bytes each exceed half the budget; charging the declared size again lets thesecond through.
the_source_budget_bounds_what_is_retainedandan_abandoned_task_returns_its_source_budgetpin the limit and its release.a_refused_source_tells_the_agent_rather_than_claiming_a_read_failurepins the wording split.a_rejection_reason_cannot_restructure_the_prompt_linefeeds a reason carrying\n[System]:and abidi override; interpolating verbatim fails it.
a_reason_made_only_of_stripped_characters_still_reads_as_a_reasoncovers the empty-after-strippingcase, so the line never degrades into a dangling dash.
The limits table in
docs/inbound-attachments.mdis corrected in the same push: it described thebudget as measured against advisory declared sizes, which this change makes false.
Changes since the review at
7c0bc5f3One finding, mine, introduced with the budget itself. Fixed. The budget covered the source buffers and
nothing built from them, so it stopped bounding memory at the point where the memory actually grew.
Three distinct gaps, all real:
Fix: the source moves into its block instead of being cloned.
assemble_attachment_blockstook&[Result<Vec<u8>, _>]and opened withbytes.clone(), putting a second copy of every attachmentoutside the reservation. It now takes the sources by value and moves each one into its block. The
copy is gone by construction rather than by convention: the signature no longer offers a borrow to
clone from.
Fix: the reservation covers the block, not just the source. An image holds its source and the
base64 encoding of that source at the same time, and base64 spends four characters on every three
bytes, so the peak was about 2.3x what was charged. A text file holds its source and the code block
wrapping it. Reservations are now
source + inline_payload_bytes(type, source), so the peak is whatgets charged. Audio and video are unchanged and charge nothing extra: they carry a URL and metadata
whatever their source weighs.
Fix: the queued payload is bounded. The guards were scoped to the assembly branch, so the
reservation was released the moment the blocks existed, and those blocks then sat in the dispatcher
queue. That queue is bounded by
max_buffered_messages(10 per thread) and not by bytes, so nothingbounded it in size at all. Guards are now held for the whole task, and a per-message cap bounds the
handover: 24 MiB of inlined payload, past which the attachment is described rather than inlined,
taking the same
not deliveredline the other limits produce.The two byte limits deliberately cover different lifetimes, which the limits table in
docs/inbound-attachments.mdnow states: the 256 MiB budget covers preparation and is returned atsubmit, the per-message cap covers what the dispatcher then holds.
inlining more than 24 MiB now describes the overflow instead of inlining it; nothing else changes
shape. Rollback: revert the commit.
Tests.
an_image_reserves_what_its_encoded_block_will_holdgives an image exactly enough budgetfor its source and asserts it is still refused, with a positive control at the full bound so the test
cannot pass vacuously; charging the source alone fails it.
a_near_limit_image_is_described_rather_than_inlinedsets the cap to exactly one encoded block andasserts the second image comes back as a
not deliveredline rather than an image; dropping the capfails it.
the_inline_budget_admits_exactly_the_limitpins the arithmetic and the saturatingboundary.
One caveat stated rather than claimed away: the guard-lifetime change is a scoping change at a single
call site inside the spawned per-event task, and is verified by reading rather than by a test. The
unit tests drive
read_attachment_sourcesandassemble_attachment_blocksdirectly, so they cannotobserve where that call site binds its guards.
Changes since the review at
6e07f6fbFour findings. Three fixed, one fixed in part with the remainder stated plainly below rather than
implied away.
Fix (F1, memory half): a shed attachment releases its bytes. Shedding described the attachment
from metadata but left the payload in the event the spawned task then captured, so an event shed
precisely because the broker was over its limit went on holding those bytes through an unbounded wait
for its turn. The limit named memory and bounded none of it.
shed_attachment_payloadnow builds thedescription and drops the bytes on the receive path, before any task can capture them.
Not fixed (F1, admission half), and this is the honest part: the number of spawned tasks is still
unbounded. Bounding it means choosing one of two things the review cannot choose on the author's
behalf. Either the broker drops user messages under load, which is a product decision with its own
impact statement, or the design changes from one task per event to one consumer per thread draining a
queue, which is a redesign of this scheduler rather than a fix to it. What the shed fix does buy is
that a parked task's cost is now metadata and prompt text rather than attachment bytes, so the
failure mode is bounded in kind even where it is not bounded in count. This belongs with the scope
question already open below.
Fix (F3): the inline cap charges only what reaches the prompt. The cap ran before the text branch
decided whether a filestore would take the file, so a large text file delivered as a URL spent an
allowance the outgoing message never used, and a later image was refused for space that was in fact
free. The delivery decision is now made first, in
goes_to_filestore, and the charge follows it.Fix (F2): an upload reserves for the copy it makes.
retained_upper_boundtreated audio assource-only, but
ByteStream::from(data.to_vec())builds a second full buffer while the original isstill alive for STT. Text above the inline limit takes the same path. Both now reserve for source
plus copy, so concurrent uploads cannot sit outside the 256 MiB bound.
Fix (F4): the gateway video contract says one thing. The pre-dispatch section claimed gateway
audio and video carry URL metadata. Gateway video is rejected before Core sees it and has no branch
in the assembly
matchat all, which the Unsupported Types section already said. Only audio isclaimed now, and the surrounding text spells out which bytes each of the two byte limits counts.
message is unchanged in what the agent sees and only changes in what the broker keeps. Deployments
with a filestore will find large text files no longer consume the inline allowance, which admits
strictly more attachments than before. Rollback: revert the commit.
Tests.
a_shed_event_carries_no_attachment_bytesasserts both halves, that the agent still getsa description naming the limit and that the payload is gone; keeping the payload fails it.
a_filestore_moves_the_charge_from_the_prompt_to_the_uploadpins the delivery-mode split at theinline limit and the doubled reservation for an upload; dropping the upload charge fails it, and so
does charging externalized text against the inline cap, independently.
One test the review asked for is not here: a filestore-enabled mixed text-plus-image integration
test.
Filestoreis a live S3 client with no test double in this repo, so the reachable equivalentis the pure decision the integration test would be exercising, which is what the test above covers.
Building an S3 double is worth doing and is worth doing outside this PR.
Changes since the review at
0587c93bTwo findings, both fixed. The second is the admission bound left open last round.
Fix: the encoded input is charged and then released. An attachment can arrive as base64 on the
event rather than as a colocated path, and serde allocates that string when the event is parsed.
Nothing charged it and nothing freed it, so a 20 MiB inline attachment carried roughly 26.7 MiB of
base64 through assembly and dispatcher handoff, entirely outside the budget that claimed to bound
attachment memory. A refused attachment kept it as well, which is precisely backwards: the refusal
existed to avoid holding those bytes.
read_attachment_sourcesnow takes the input off the event withmem::takebefore decoding, on every path including both refusal paths, and the reservation coversit alongside the buffer decoded from it.
Fix: admission is bounded. Past 256 events in preparation the broker refuses the event and tells
the sender to send it again, rather than admitting work it has no way to bound.
The trade-off is worth stating rather than burying, because there was no option without one.
Bounding admission means either refusing work or stalling the socket, and stalling the socket takes
/canceldown with it, which is the exact failure this whole path was built to avoid. Refusing isvisible to the sender and recoverable by them in a way that a queue growing until the process dies is
not. The limit is a compile-time constant with no config key, consistent with the others here: 256
events in flight is a load problem to report, not a number to tune.
simultaneous preparation is far past ordinary traffic. Under sustained overload a sender now gets a
refusal notice instead of the broker silently accumulating work. No config, schema, or wire-format
change. Rollback: revert the commit.
Tests.
the_reservation_covers_the_encoded_input_and_then_frees_itgives an attachment exactlyenough budget for its decoded buffer and asserts it is still refused, then asserts the input is gone
from the event on both the refused and admitted paths; not charging the input fails it, and so does
keeping it on the event, independently.
admission_stops_at_the_limitpins the boundary.an_image_reserves_what_its_encoded_block_will_holdandan_under_reported_size_cannot_bypass_the_source_budgetwere updated to the new peak arithmetic andstill falsify what they were written for.
Correcting the previous section. The note under
6e07f6fbabove said the unbounded task countwas left to a maintainer decision. That is no longer accurate: the policy is chosen and implemented
here. The section stays as the record of what was true at that commit.
Changes since the review at
a90bb6ccFour findings, all fixed. Two of them are on the unified ingress path, which had none of the controls
the WebSocket path has accumulated.
Fix: unified ingress shares one set of limits. The bridge spawns a task per event and
process_gateway_eventconstructedSourceBudget::new(MAX_ADMITTED_SOURCE_BYTES)inside each one, soevery concurrent event could independently reserve the entire 256 MiB. A per-event budget is
arithmetically the same as no budget. The budget, a fetch semaphore, and an admission counter now
live on
GatewayIngressLimits, held by the shared event context, which is built once and cloned perevent. The refusal reply is the same one the WebSocket path sends.
Fix: control commands run before any attachment work. The unified path read and assembled
attachments before checking
/reset,/cancel, and config commands, so a/cancelcarrying a voicenote could upload to object storage and run transcription before returning without dispatching
anything. Commands are now handled first, matching the WebSocket path.
Fix: text is charged what it renders to.
String::from_utf8_lossyspends a three-bytereplacement character on every malformed byte, so a 20 MiB invalid-UTF-8 text file passed the 24 MiB
inline cap and produced roughly a 60 MiB block. Validity is checked with
str::from_utf8before theconversion, so the answer is known without allocating the expansion it predicts, and the charge
follows it. Anything charged before its bytes are read now assumes the worst case, which is the only
safe assumption available at that point.
Fix: a truncated read is a failure, not an attachment.
file.take(limit).read_to_endreturns aprefix as success, so a colocated file replaced between the metadata read and the data read was
delivered truncated with nothing to say so. The read now looks one byte past the reservation and
fails if it finds one.
concurrent load they now refuse rather than reserving without bound; a control command carrying
audio stops doing discarded work. Valid text files are charged exactly as before. No config,
schema, or wire-format change. Rollback: revert the commit.
Tests.
malformed_text_is_charged_for_what_it_renders_topins the lossy split and the worst-casepre-read charge.
a_source_that_grew_after_admission_is_not_delivered_as_a_prefixwrites a file,reserves its length, grows it, and asserts the read fails.
unified_ingress_limits_are_shared_not_per_eventandunified_admission_releases_its_slotprove aclone sees the same budget and the same admission counter. Each of the three corresponding falsifiers
fails only its own test.
One gap, stated rather than glossed: the control-command ordering has no runtime test. Proving it
needs a
GatewayEventContext, which means adapter, dispatcher, and router doubles that this module'stest suite does not have, and building the first such fixture is a larger piece of work than the fix
it would cover. The fix itself is a reordering in which each command branch returns before any
attachment code is reachable.
At a Glance
The transcript is unchanged in content and position. The audio block is additive. The
Slack video block keeps its shape and gains a working
urlplus anoteline; see theimpact statement under Why this approach?.
Discord video is untouched: its URL already needs no credentials, so it passes
Noneforthe note and its emitted text stays byte-identical (asserted by
video_attachment_block_omits_note_line_when_none).Prior Art & Industry Research
Both reference projects treat inbound audio as a transcription source only: the agent sees
text, never the file. OpenClaw goes further and deletes the media after the preflight pass.
Neither has a "STT is off, so hand the agent the file instead" path, so this is a gap rather
than a solved problem with a house style to copy.
extensions/slack/src/monitor/message-handler/preflight-audio.ts,docs/nodes/audio.md)[Audio]blockresolveSlackPreflightAudioTranscriptreturns{ transcript, mediaIndex }, no path or bytesnulland the message falls back to text-only mention detection.discardSlackPreflightMediathenfs.rms the downloaded media, so the file is unrecoverableagent/transcription_provider.py,agent/transcription_registry.py){"success": true, "transcript": ...}; there is no field for the source file{"success": false, "transcript": "", "error": ...}. No fallback that surfaces the audio itselfmedia::is_audio_mimebranch in each adapterdebug!log (gateway); file dropped[Audio attachment]block with a fetchable URLOpenClaw: transcript only, and it actively destroys the file.
preflight-audio.tsruns a per-channel audio preflight before mention detection;
resolveSlackPreflightAudioTranscriptreturns{ transcript, mediaIndex }with no path orbytes, and
docs/nodes/audio.mdconfirms the message body is replaced with an
[Audio]block. When transcription fails itreturns
nullanddiscardSlackPreflightMediacallsfs.rmon the download, so the file isunrecoverable. There is no "STT is off, hand over the file" path.
Hermes Agent: transcript only, with no field for the source file.
agent/transcription_provider.pydefines a provider registry over local/Groq/OpenAI backends behind one envelope; success is
{"success": true, "transcript": ...}and failure is{"success": false, "transcript": "", "error": ...}. Neither shape can carry the audio, sothere is no fallback that surfaces it.
Other references: openab's own #738 is the closest prior art and is in-repo. It resolved
the same question for binary files by rejecting raw platform URLs in favour of filestore
presigned URLs;
docs/filestore.mdrecords why, naming PR #1346's raw-URL hint and itslimitations. This PR applies that resolution to audio and Slack video rather than reopening it.
What we learn: the transcript-only shape is the industry default, and it is precisely why both
projects must treat "STT unavailable" as "message unavailable". Decoupling the file from the
transcript removes that coupling, and openab is well placed to do it because #738 already built
the credential-free URL mechanism that OpenClaw and Hermes lack.
Proposed Solution
An
[Audio attachment]block is emitted for every audio attachment, on every adapter,independent of the STT setting. Shape matches the existing
[Image attachment]/[Video attachment]blocks:Which URL the agent gets follows the #738 precedent, filestore first with a per-platform
fallback:
attachment.url, note records the ~24h CDN expiryurl_private_download, note records that a Bearer bot token is requiredurlline, note points at filestoreThe gateway asymmetry is structural, not an oversight:
openab-gateway'sAttachmentcarries base64
dataor a colocatepathand never a platform URL, because the gatewayalready consumed the platform credential during download. The colocate path is deliberately
not exposed to the agent either, since
store.rsevicts it after 120s and it would be adead path by the time most skills fetch it.
Slack video takes the same route. It now tries filestore first and falls back to
url_private_downloadwith a note naming the bearer-token requirement, so the agent isnever handed a link without being told what it needs. Discord video is unchanged.
Implementation:
media::audio_attachment_block()andmedia::video_attachment_block(), two sharedbuilders over one
sanitize_attachment_meta()helper. Filename and MIME are sanitised(control characters stripped, capped at 200/100 chars) before entering the prompt, per the
same rule the filestore hints follow. The video builder replaces a private copy in
discord.rsand an inlineformat!inslack.rsthat had already drifted apart.media::download_and_presign_attachment()andmedia::upload_bytes_and_presign(), forcallers that build their own block (URL-based adapters and the gateway's in-memory bytes
respectively). Both return the presigned URL paired with its note, built by one private
presigned_note(). The five call sites would otherwise each rebuild the same"presigned URL, expires in N minutes"wording, which is the drift this PR is removingelsewhere.
download_and_upload_any_file()is refactored onto a shareddownload_and_presign_any_file()core returning
Result<_, PresignError>. Its three degraded hint strings are preservedverbatim, so the feat: support non-image binary file attachments inbound (PDFs, office docs, video) on Slack/Discord #738 PDF/ZIP path is byte-identical.
"audio" if stt_config.enabledarms plus their fallthrough arms collapseinto one
"audio"arm, so the passthrough cannot be forgotten on one of the two paths.Not changed: transcript text and its pairing with the file,
EchoEntrybehaviour, the🎤 reaction, Discord video, and every remaining attachment branch.
Why this approach?
Why unconditional rather than behind a config flag. No attachment class in openab is
gated by a per-type toggle:
FilestoreConfig(config.rs:149) has noenable_*field, andno per-attachment-type passthrough setting exists anywhere in the config tree. The only knob has always been whether a filestore is
configured, which this change honours. A toggle here would make audio the one class that can
be configured to silently discard a user's file, which is the behaviour #251 was filed to fix.
Impact statement (behaviour change on upgrade). Two changes, of different kinds.
Audio is purely additive. Every deployment that receives audio sees one additional text
block per audio attachment, roughly 40 to 60 tokens, the same order as the existing
[Image attachment]block. Nothing that exists today changes.Slack video is a modification of an existing block, and is the only place where output
a deployment already depends on will differ. Concretely:
url:isurl_private_download, 403s for the agenturl:is a presigned URL that resolves, plus anote:lineurl:isurl_private_download, no explanationnote:naming the bearer-token requirementmainwould emit, asserted by testAnything parsing the block by line prefix keeps working, since
note:is appended afterurl:and the four existing lines keep their order and names. The risk is a consumer thatpattern-matches the URL for a
files.slack.comhost, which would now see the filestorehost instead. That consumer is currently matching on a URL it cannot fetch, so this is
the failure surfacing rather than a new one. There is no config, schema, or wire-format
change either way, no new failure mode (every failure path falls back to a less specific
block rather than an error), and rollback is a revert with no migration.
Dual-path validation is under Validation: the suite is green both with and without the
filestorefeature, and the no-filestore path is the one that exercises the platform-URL fallback.
On the opt-out that the behaviour-change policy asks for. Raising it rather than hoping it
goes unnoticed: this PR ships no new flag. The reasoning is that the policy's stated purpose
is that "every existing deployment switches behavior on upgrade", and here the Slack video
half is a bug fix rather than a default flip. The behaviour it replaces is a URL that returns
403 to the agent every time, so an opt-out would be an option to keep a link that never
worked, and no deployment can be depending on it succeeding.
What does exist, and is validated, is the degradation path: a deployment with no filestore
keeps
url_private_downloadexactly as today and gains only the additivenote:line. Thatis the "old behaviour is still reachable" property the policy is protecting, and the suite is
green in that configuration. If a maintainer reads the rule more strictly than that, the
cheapest resolutions are to split the video half into its own PR where it can carry a flag,
or to gate it behind
[slack] video_via_filestoredefaulting tofalseand flip in afollow-up. Say which you prefer and it will be done rather than argued.
Accepted limitation: a second download on filestore deployments. With STT enabled and a
filestore configured, Discord and Slack now download the file twice, once inside
download_and_transcribeand once to stream into S3. The cause is thatdownload_and_transcribeowns its download;stt::transcribeitself already takes bytes, asthe gateway path shows, so the fix is to download once and feed both. That is a change to the
STT path and this PR deliberately does not touch STT, per one-concern-per-PR. Named follow-up:
have the Discord/Slack audio branch download once and call
upload_bytes_and_presign+stt::transcribeon the same buffer, which also lets the two adapters share the gateway'sshape. The gateway path already has no double download.
On the error shape of the two new presign helpers. They return
Result<(String, String), AudioStoreError>, not theOptiontheirmainsiblings use.An earlier revision of this PR matched the neighbours and was wrong to: collapsing a size
refusal, a failed platform download and a failed upload into one
Noneis what let thegateway report a configured-but-broken filestore as a missing one. The typed error is what
lets each note name the cause that actually occurred.
download_and_upload_any_filekeepsits
Optionsignature, so the#738path is untouched.Alternatives Considered
Metadata block with the platform URL only, no filestore (PRD "Option A"). Smallest diff,
about 5 lines per adapter, but it does not work where it matters most. Slack's
url_private_downloadneeds a Bearer token, and putting a bot token in the agent's prompt isthe security regression already argued against in #738. It also cannot serve the gateway,
which holds no platform URL at all.
docs/filestore.mdalready records this judgement: it names PR #1346's raw-URL hint and listsexactly these limitations (Discord CDN URLs expiring in ~24h, Slack's
url_private_downloadrequiring "a Bearer token the agent does not have"). This PR does not relitigate that. The
filestore presigned URL is the primary path on all three adapters, and the platform URL appears
only as a labelled degradation for deployments with no filestore, where the alternative is the
current behaviour of dropping the file silently. The
notefield states the applicable caveatverbatim, so an agent can tell a fetchable URL from one it will be refused, rather than
discovering it through an unexplained 403 or an HTML login page.
Filestore upload only (PRD "Option B"). Clean and credential-free, and the right answer
when a filestore exists, but it silently does nothing for deployments without one. That is the
majority of small self-hosted setups, and they are exactly the ones that would notice audio
still vanishing.
Chosen: filestore with a per-platform fallback (PRD "Option C"). Presigned URL where
available, honest platform URL plus a caveat note where not, metadata only where neither
exists. Every deployment gains something and none regresses.
Exposing the gateway's colocate path. Tempting, since the agent shares the filesystem in
colocate mode and a local path suits file-processing tooling better than a URL. Rejected on
openab-gateway/src/store.rs:37:TTL_SECS = 120, with a background task evicting on thatschedule. A path that is usually dead on arrival would invite skills to build on it.
Download the audio to disk and hand over a path (the shape originally proposed in #254 and
#738). Superseded by #738's resolution, which chose presigned URLs. Following the pattern that
landed keeps one mechanism instead of two.
Validation
cargo checkpassescargo testpasses (including new tests)cargo clippycleanBase:
main@53061d69, rustc/clippy 1.91.0. Every command below is oneci.ymlruns forcrates/**, and each was run twice, once on this branch and once in a detached worktreeat the unmodified base, so pre-existing failures are separated from regressions rather than
asserted to be pre-existing.
Both re-measured against
upstream/mainat53061d69, so747 - 707 = 40is the currentnet delta. The single failure is
secrets::tests::resolve_exec_nonzero_exiton both sides,confirmed by name; it is a macOS-only
/bin/falsefailure untouched here. The delta was takenby diffing
cargo test -- --listbetween the two trees rather than by counting#[test]attributes: 41 names added, 1 removed (
video_attachment_block_includes_actionable_metadata,moved out of
discord.rsintomedia.rsalongside the function it covers). The 41 are theaudio and video block tests, the
audio_mimeclassifier tests including the MIME-casing andexplicit-non-audio boundaries, the gateway outcome and store-failure tests, the four
gateway-arm tests, the two per-adapter store-outcome row tests, the
presigned_ttlcap andlifetime-rendering tests, the upload-API compatibility test, the sanitiser and
prompt-forging tests, the video parity table with its two truncation-boundary companions,
the post-response failure-classification test, and the two attachment-assembly tests.
Scoped clippy, five feature configurations (dual-path validation for the impact statement
under Why this approach?). Narrowing the new items to
pub(crate)is what made the extra threeworth running:
pubitems are exempt from the dead-code lint andpub(crate)ones are not, sothe configurations with no
filestoreand with no adapter feature each turned up unreachablecode that a
pubsurface had been hiding. All five are clean:Two pre-existing failures on
main, both reproduced on the unmodified base. Neither istouched by this PR and neither is fixed by it, per one concern per PR.
cargo clippy --workspace -- -D warningsfails oncrates/openab-core/src/pre_seed.rs:471with
collapsible_else_if, a lint that newer clippy applies to code this PR does not touch.The base worktree exits
101with the identical error.cargo clippy --workspace --features unified -- -D warningsreports 2 errors. The errorset is byte-identical between base and this branch, compared by diffing the sorted
^errorlines from both runs.
cargo fmt --checkis not a gate for this PR. It exists only inci-openab-agent.yml,which is path-filtered to
openab-agent/**andcrates/openab-mcp/**and runs withworking-directory: openab-agent;ci.yml, the workflow that coverscrates/**, has no fmtstep.
For the record rather than as a claim of cleanliness: workspace-wide
cargo fmt --checkunderrustfmt 1.8.0 reports 488 diffs on the unmodified base and 482 on this branch. Every
line this PR adds was hand-formatted to rustfmt's shape, verified by fingerprinting each
Diff inhunk on both sides and confirming that no hunk is unique to this branch. The net -6is pre-existing hunks this PR happens to rewrite out of existence, most of them inside the two
gateway attachment loops that collapsed into one helper. No crate-wide
cargo fmtwas run, because it would reformat several hundreduntouched lines across files this PR does not own, for no CI benefit.
The
1 failedin every run above issecrets::tests::resolve_exec_nonzero_exit, a thirdpre-existing failure and an environmental one. It shells out to
/bin/false, which recentmacOS does not ship, so the error text is "not found" rather than the asserted "exited with".
The base worktree fails it identically. It passes on Linux CI, which is where this matters.
New tests (
crates/openab-core/src/media.rs)The audio four cover the presigned-URL shape, the gateway no-URL shape (asserting the
url:line is absent rather than empty), prompt-injection hygiene (a filename carrying\nurl: ...must not forge a second field), and the empty-MIME fallback.The video four are the regression guard for the impact statement under Why this approach?.
video_attachment_block_omits_note_line_when_noneis the load-bearing one: it asserts theexact full string, not
contains, so any future drift in Discord's output fails thebuild.
video_attachment_block_includes_actionable_metadatais the test that previouslylived in
discord.rs, moved unchanged alongside the function it covers.These are pure-function tests on the block builders, matching how the adapters are covered
today:
discord.rstests the image block's format string the same way, and the adaptermodules have no mock-transport harness to extend.
Docs.
docs/inbound-attachments.mdgains the block format, the per-platform URL table,and the gateway rationale; its support matrix now reads "file + STT".
docs/stt.mdno longerclaims audio is "silently skipped" when STT is disabled, on both the config table row and the
disabled-behaviour paragraph. Anchors verified against the rendered headings.
Live run. This branch's code has been running on a private 9-agent Slack/Discord
deployment since 2026-07-27, against a real Cloudflare R2 bucket through the
S3-compatible API. The filestore chain is confirmed from its logs end to end: a 1.0MB
attachment streamed up (
filestore streaming upload complete ... size=1009827), waspresigned with
X-Amz-Expires=3600, and the agent then fetched it withcurl -sL -oagainst that URL, unprompted, from the block alone. Zero upload, presign, or timeout
errors over the period.
Manual testing: both new paths, end to end on Slack. Object keys and signature
parameters are redacted in the transcripts and masked in the screenshots, because the
bucket is publicly served in this deployment and the key alone is enough to fetch the
object. The screenshots also mask the audio's transcript text, which is unrelated private
content.
Video. A 1MB
Big_Buck_Bunny_360_10s_1MB.mp4posted to a Slack thread. The agentreceived the
[Video attachment]block and, unprompted, ran:It then probed the duration with
ffprobe, extracted 8 frames withffmpeg, read them,and identified the clip as the opening forest shot of Blender's Big Buck Bunny. This is
the case that fails on
main: the URL would have beenurl_private_downloadand thecurlwould have returned 403 with no note explaining why.Audio. A 23-second, 202KB
.m4aposted to the same workspace. The agent received the[Audio attachment]block and ran:then ran a transcription skill against the downloaded path. The message carries both the
🎤 STT reaction and the passthrough, so the transcript and the file block coexisted rather
than one replacing the other, which is the property this PR exists to establish.
Not covered. The
media::video_attachment_blockextraction and the note-pairingrefactor are both newer than the build that produced the runs above; that build carried the
same routing logic inline. Both refactors are behaviour-preserving and the exact-string test
video_attachment_block_omits_note_line_when_noneis what asserts it, but the refactoredform has not itself been exercised in production.
The gateway adapters were not manually tested; no Telegram/Feishu/LINE/Google Chat
deployment was available. Their audio path is covered by unit tests and by the shared
builder the Slack and Discord paths exercise.
There is also no mock-server integration test for the filestore upload itself, since
openab-core'ssrc/has no precedent to follow (the#[ignore]integration tests live inthe agent crates).