Skip to content

feat(tts): add Microsoft Edge TTS provider#168

Merged
grinev merged 3 commits into
grinev:mainfrom
hagope:feat/edge-tts-provider
Jul 6, 2026
Merged

feat(tts): add Microsoft Edge TTS provider#168
grinev merged 3 commits into
grinev:mainfrom
hagope:feat/edge-tts-provider

Conversation

@hagope

@hagope hagope commented Jun 21, 2026

Copy link
Copy Markdown
Contributor

Summary

Adds Microsoft Edge TTS as a fourth TTS provider (TTS_PROVIDER=edge). It uses Microsoft Edge's online Read Aloud service via WebSocket — no API key, account, or TTS_API_URL required; only outbound HTTPS/WebSocket access to speech.platform.bing.com.

Implementation ports the protocol from the Python edge-tts reference.

How it works

  • DRM token (Sec-MS-GEC): SHA256 of Windows file-time ticks (rounded to 5 min) + trusted client token. Includes a single 403 retry that corrects clock skew from the server's Date header.
  • SSML framing: prosody rate/volume/pitch, with 4096-byte UTF-8-safe chunking that never splits multi-byte characters or XML entities. Attribute values (voice/rate/volume/pitch) are XML-escaped.
  • Per-chunk connections: each SSML chunk is synthesized over its own WebSocket connection (matching upstream edge-tts — the service does not reliably accept a second SSML turn on one socket), and the resulting MP3 buffers are concatenated. A close before turn.end always rejects; partial audio is never returned as success.
  • Shared deadline: one 60s budget (TTS_REQUEST_TIMEOUT_MS) bounds the whole synthesis across all chunks, matching the other providers' timeout semantics.
  • WebSocket protocol: binary audio frames use a 2-byte big-endian header-length prefix (length value includes the trailing \r\n), so audio starts at offset 2 + headerLength.
  • Output format: audio-24khz-48kbitrate-mono-mp3.

Changes

  • New: src/app/services/edge-tts.ts + tests/app/services/edge-tts.test.ts
  • Wire 'edge' provider into src/config.ts and src/app/services/tts-service.ts
  • isTtsConfigured() returns true for edge (no credentials needed)
  • Document provider in .env.example and PRODUCT.md
  • Adds ws as a runtime dependency (Node 20+ target; the global WebSocket is only stable from Node 22+)

Configuration

TTS_PROVIDER=edge
TTS_VOICE=en-US-EmmaMultilingualNeural

Verification

  • npm run build / npm run lint / npm test all pass (1088 tests)
  • Verified end-to-end against the real Edge TTS service, including the multi-chunk path: an 8.9 KB input split into 3 chunks produced a valid concatenated MP3 (48 kbps, 24 kHz mono) in ~6s, consistent across 5 consecutive runs

@grinev

grinev commented Jul 1, 2026

Copy link
Copy Markdown
Owner

@hagope thanks for the PR! The overall integration looks clean, and the config/provider wiring makes sense. Build, lint, and tests pass after installing the new dependency.
One issue I’d like to address before merging: multi-chunk Edge synthesis appears to be problematic.
The current implementation opens one WebSocket for all chunks and sends the next SSML message after turn.end:

  • streamSynthesis() calls attemptSynthesis(chunks, params) once
  • attemptSynthesis() creates a single WebSocket
  • after turn.end, it sends the next chunk on the same socket

However, the upstream edge-tts implementation opens a separate WebSocket synthesis stream per chunk. I also smoke-tested this against speech.platform.bing.com: short text works, but longer text that gets split into multiple chunks times out after 60s.

This can affect real bot usage because the bot limits TTS input by characters (MAX_TTS_INPUT_CHARS = 4000), while Edge splits by UTF-8 bytes (4096). Non-ASCII text can easily exceed 4096 bytes while still being under 4000 characters.
There is also a related edge case: if the socket closes after the first chunk, the current close handler may return partial audio as success because it checks only audioReceived, not whether all chunks were processed.

Could you please adjust the implementation to synthesize each chunk using a separate WebSocket connection, then concatenate the resulting MP3 buffers? Alternatively, at minimum, the code should not treat close as success unless all chunks have completed.

hagope and others added 3 commits July 5, 2026 17:00
Adds a fourth TTS provider that uses Microsoft Edge's online Read
Aloud service via WebSocket. No API key, account, or TTS_API_URL
required; only outbound HTTPS/WebSocket access to
speech.platform.bing.com.

Implementation ports the protocol from the Python edge-tts reference
(https://github.com/rany2/edge-tts):
- Sec-MS-GEC DRM token: SHA256 of Windows file-time ticks (rounded to
  5 min) + trusted client token, with single 403 retry that corrects
  clock skew from the server Date header
- SSML framing with prosody rate/volume/pitch and 4096-byte UTF-8-safe
  chunking that never splits multi-byte chars or XML entities
- WebSocket message handling: binary frames use a 2-byte big-endian
  header-length prefix (length includes the trailing CRLF), audio
  starts at offset 2 + headerLength; text frames signal turn.end per
  chunk

Adds ws as a runtime dependency (Node 20+ target; global WebSocket is
only stable from Node 22+).

- New: src/app/services/edge-tts.ts + tests
- Wire 'edge' provider into config.ts and tts-service.ts
- isTtsConfigured() returns true for edge (no credentials needed)
- Document provider in .env.example and PRODUCT.md
The service does not reliably accept a second SSML turn on the same
socket: multi-chunk input (possible because the bot's 4000-char limit
can exceed Edge's 4096-byte chunk limit for non-ASCII text) hung until
the 60s timeout. Open a fresh connection per chunk, as the upstream
edge-tts client does, and concatenate the resulting MP3 buffers.

A close before turn.end now always rejects instead of returning
partial audio as success.

Also stub TTS_PROVIDER/TTS_MODEL in the config default-values test so
ambient environment variables no longer leak in.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Keep an error sink on the WebSocket during teardown: in ws 8.x,
  close() on a still-connecting socket emits 'error' on the next tick,
  and with listeners removed that was an uncaught exception that
  crashed the process on every 403 or pre-connect timeout.
- Bound the whole synthesis (all chunks) by one shared deadline wired
  from tts-service's TTS_REQUEST_TIMEOUT_MS, matching the other
  providers instead of allowing 60s per chunk.
- Fix binary-frame bounds check: header occupies [2, 2+headerLength),
  so guard against 2 + headerLength > buf.length.
- Escape voice/rate/volume/pitch when interpolated into SSML
  attributes so a TTS_VOICE with ' or & fails loudly, not as
  malformed XML.
- Drop the unreachable edge branch in getNotConfiguredMessage
  (isTtsConfigured is always true for edge) and the dead throw after
  the 403 retry; write the clock-skew update as an absolute
  assignment; reuse EDGE_DEFAULT_VOICE in tts-service.
- Simplify removeIncompatibleCharacters to a regex replace,
  jsDateString to a toUTCString reorder, and the UTF-8 split probe to
  a continuation-byte check.
- Add tests: SSML attribute escaping, fresh-connection 403 retry, and
  the shared cross-chunk deadline.

Verified end-to-end against the live service: 3-chunk (8.9KB) input
synthesizes a valid MP3 in ~6s across 5 consecutive runs.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@hagope hagope force-pushed the feat/edge-tts-provider branch from 430ba7e to 1332453 Compare July 6, 2026 00:29
@hagope

hagope commented Jul 6, 2026

Copy link
Copy Markdown
Contributor Author

Thanks for the thorough review and the smoke test — you were right, the shared-socket design was the problem. Fixed and rebased onto main (the src/config.ts conflict is resolved).

Multi-chunk: each chunk now gets its own WebSocket connection, matching upstream edge-tts, and the MP3 buffers are concatenated in order. The partial-audio edge case is fixed too: a close before turn.end now always rejects — audio is never returned as success unless the turn completed.

A few more hardening fixes that came out of re-reviewing the module:

  • Timeout semantics: one shared 60s deadline (TTS_REQUEST_TIMEOUT_MS) now bounds the whole synthesis across all chunks, so a multi-chunk request can't hold the reply pipeline for 60s × N like the per-connection timers would have allowed.
  • Crash on the 403 path: finish() removed all listeners before ws.close(); in ws 8.x, closing a still-connecting socket emits 'error' on the next tick, which with zero listeners is an uncaught exception. An error sink now stays attached during teardown.
  • Off-by-two in the binary-frame bounds check (2 + headerLength > buf.length).
  • SSML attribute values (voice/rate/volume/pitch) are now XML-escaped.
  • Removed the unreachable edge branch in getNotConfiguredMessage() and other dead code.

Verified end-to-end against the live service: an 8.9 KB input → 3 chunks → valid concatenated MP3 in ~6s, consistent across 5 consecutive runs (previously this exact case hit the 60s timeout). New tests cover per-chunk connections, ordered concatenation, partial-audio rejection, the fresh-connection 403 retry, and the shared deadline.

@grinev

grinev commented Jul 6, 2026

Copy link
Copy Markdown
Owner

@hagope thanks for contribution!

@grinev grinev merged commit f096c68 into grinev:main Jul 6, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants