AmAudio::decode: bound decoded PCM size against back buffer (remote heap overflow)

[hecko]

Bug

AmAudio::decode() (core/AmAudio.cpp) passes the encoded RTP payload size straight to codec->decode() without checking how much PCM the codec will produce:

s = (*codec->decode)(samples.back_buffer(), samples, s, ...);

The decoder writes its PCM16 output into samples.back_buffer(), which is a fixed AUDIO_BUFFER_SIZE (8192-byte) half of the DblBuffer (unsigned char samples[AUDIO_BUFFER_SIZE * 2], core/AmAudio.h).

AmRtpAudio::receive() reads up to AUDIO_BUFFER_SIZE bytes of RTP payload and hands it to decode(). An expanding codec — GSM (~10×), G.722 (4×) — fed a large payload emits far more than AUDIO_BUFFER_SIZE bytes of PCM and writes past the end of the DblBuffer. This is a remotely-triggerable heap buffer overflow reachable on an established call by a peer sending an oversized codec payload.

Fix

Predict the decoded sample count with bytes2samples() and reject the frame when the resulting PCM size would exceed the buffer, before invoking the codec:

unsigned int out_size = PCM16_S2B(bytes2samples(size));
if(out_size > AUDIO_BUFFER_SIZE){
  ERROR(...);
  return -1;
}

Normal RTP frames (≤30 ms) decode to well under 8192 bytes, so legitimate traffic is unaffected; only abnormally large payloads are dropped.

Provenance

Backported from yeti-switch/sems, which added the same pre-decode buffer-size guard in AmAudio::decode(). sems-server does not have yeti's optional codec frames2samples() hook, so this uses the existing bytes2samples() path — equivalent to yeti's fallback branch. Thanks to the yeti-switch maintainers for the fix.

---

Generated by Claude Code

[Copilot]

Pull request overview

This PR adds a defensive guard in AmAudio::decode() to prevent decoders from writing more PCM16 output into the fixed-size DblBuffer back buffer than it can hold, addressing a remotely reachable buffer overflow risk when decoding oversized RTP payloads with expanding codecs.

Changes:

• Add a pre-decode output-size check using bytes2samples() to reject frames whose decoded PCM16 would exceed AUDIO_BUFFER_SIZE.
• Add detailed in-code commentary explaining the overflow scenario and rationale for the guard.
• Minor whitespace cleanup near the end of decode().

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.