Skip to content

Do not forward the audio codec config buffer as an encoded frame - #2202

Open
RomanHerbstmann wants to merge 1 commit into
pedroSG94:masterfrom
RomanHerbstmann:audio-drop-codec-config-buffer
Open

RomanHerbstmann wants to merge 1 commit into
pedroSG94:masterfrom
RomanHerbstmann:audio-drop-codec-config-buffer

Conversation

@RomanHerbstmann

Copy link
Copy Markdown
Contributor

Problem

MediaCodec emits the audio codec configuration as its first output buffer, flagged with BUFFER_FLAG_CODEC_CONFIG. For AAC that buffer is the 2-byte AudioSpecificConfig.

BaseEncoder.processOutput forwards every buffer that checkBuffer accepts:

if (checkBuffer(byteBuffer, bufferInfo)) sendBuffer(byteBuffer, bufferInfo);

VideoEncoder.checkBuffer keeps SPS/PPS out of that path, but AudioEncoder.checkBuffer only validates the timestamp, so the config buffer is forwarded like a regular encoded frame. It then reaches

  • the senders, which transmit it as a tiny bogus AAC frame, and
  • recordAudioAndroidMuxerRecordController.writeMediaMuxer.writeSampleData, which stores it as a media sample.

Impact

Recorded MP4 files start with a 2-byte audio sample that is identical to the track's extradata. FFmpeg and VLC skip it with Input buffer exhausted before END element found, but Chrome aborts playback of the entire file:

PIPELINE_ERROR_DECODE: Failed to send audio packet for decoding:
{timestamp=0 duration=103084 size=2 is_key_frame=1 encrypted=0}

Verified on a 1.4 GB, 66-minute recording produced from an RTMP stream: the first audio packet is 12 10 (AAC-LC, 44.1 kHz, stereo), byte for byte the same as the track's extradata. Remuxing the same file with that single packet dropped makes it play in Chrome; nothing else changes.

Fix

Reject buffers carrying BUFFER_FLAG_CODEC_CONFIG in AudioEncoder.checkBuffer, mirroring how video keeps SPS/PPS out of the frame path.

This is safe because the configuration is derived independently on every path:

  • RTMP: AacPacket.sendAudioInfo(sampleRate, isStereo, codec) builds AacAudioSpecificConfig from sample rate and channel count.
  • RTSP: SdpBody signals it from sample rate and channel count.
  • Recording: the muxer track is created from the MediaFormat passed to setAudioFormat in onFormatChanged, which already carries csd-0.

The change covers every MediaCodec-based audio codec (AAC, HE-AAC, Opus). G711 uses the software path in BaseEncoder.processG711() and has no codec-config buffer, so it is unaffected. The filter is deliberately kept in AudioEncoder rather than BaseEncoder, because video still relies on the config buffer for manual SPS/PPS extraction when formatChanged arrives late.

Tests

Added encoder/src/test/java/com/pedro/encoder/AudioEncoderCheckBufferTest.kt with two cases: a buffer flagged as codec config is rejected, a normal frame with a valid timestamp is accepted. ./gradlew :encoder:testDebugUnitTest passes 24/24.

Note for anyone writing further tests in this module: it sets unitTests.isReturnDefaultValues = true, so MediaCodec.BufferInfo.set(...) is a no-op stub and the fields have to be assigned directly.

MediaCodec emits the audio codec config (for AAC the AudioSpecificConfig) as its
first output buffer, flagged with BUFFER_FLAG_CODEC_CONFIG. AudioEncoder.checkBuffer
only validated the timestamp, so that buffer was forwarded like a regular frame:
senders transmitted it as a tiny bogus AAC frame and AndroidMuxerRecordController
wrote it into the recording as a sample.

The resulting MP4 files start with a 2-byte audio sample that is identical to the
track extradata. FFmpeg and VLC skip it, but Chrome aborts playback of the whole
file with PIPELINE_ERROR_DECODE.

Dropping the buffer is safe because the configuration is derived independently on
every path: RTMP builds the AudioSpecificConfig in AacPacket.sendAudioInfo from
sample rate and channel count, RTSP signals it through the SDP body, and the muxer
takes it from the MediaFormat passed to setAudioFormat, which already carries csd-0.
This mirrors how VideoEncoder keeps SPS/PPS out of the frame path.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant