Raw audio decoding of video with Libav is chopped

Question

I'm currently using libav to extract the audio stream of a video into a raw PCM file.

This code works fine for mp3 but when I try with a mp4 video, the raw format imported on Audacity show stranges regular descending lines between 0 and -1.

Here is my implementation.

#include <stdio.h>
#include <stdlib.h>
#include <fcntl.h>
#include <unistd.h>

#include <libavcodec/avcodec.h>
#include <libavformat/avformat.h>
#include <libswresample/swresample.h>

int decode_raw(AVFormatContext *format_ctx)
{
    AVCodec *codec = NULL;
    AVCodecContext* codec_ctx = NULL;
    AVFrame* frame = NULL;
    AVPacket packet;
    int stream_idx = av_find_best_stream(format_ctx, AVMEDIA_TYPE_AUDIO,  -1, -1, &codec, 0);
    int res;

    if (stream_idx < 0) {
        printf("Could not find stream.\n");
        return (1);
    }

    if ((codec_ctx = avcodec_alloc_context3(codec)) == NULL) {
        printf("Could not allocate codec context.\n");
        return (1);
    }

    if (avcodec_parameters_to_context(codec_ctx, format_ctx->streams[stream_idx]->codecpar) < 0) {
        printf("Could not setup codec context parameters.\n");
        return (1);
    }

    // Explicitly request non planar data.
    codec_ctx->request_sample_fmt = av_get_packed_sample_fmt(codec_ctx->sample_fmt);

    if (avcodec_open2(codec_ctx, codec, NULL) != 0) {
        printf("Could not open codec.\n");
        return (1);
    }

    if ((frame = av_frame_alloc()) == NULL) {
        printf("Could not alloc frame.\n");
        return (1);
    }

    av_init_packet(&packet);

    int fd = open("raw", O_CREAT | O_WRONLY | O_TRUNC);

    // Decode frames.
    while ((res = av_read_frame(format_ctx, &packet)) == 0) {
        // Does the packet belong to the correct stream?
        if (packet.stream_index != stream_idx) {
            av_packet_unref(&packet);
            continue;
        }

        // We have a valid packet => send it to the decoder.
        if ((res = avcodec_send_packet(codec_ctx, &packet)) != 0) {
            printf("Failed to send packet: %d.\n", res);
            break;
        }

        av_packet_unref(&packet);
        res = avcodec_receive_frame(codec_ctx, frame);

        if (res == AVERROR(EAGAIN) || res == AVERROR_EOF)
            break;
        else if (res < 0) {
            printf("Failed to decode packet: %d.\n", res);
            return (1);
        }

        write(fd, frame->extended_data[0], frame->linesize[0]);
    }

    close(fd);
    av_frame_free(&frame);
    avcodec_close(codec_ctx);
    avcodec_free_context(&codec_ctx);
    return (0);
}

int main(int argc, char **argv)
{
    AVFormatContext *av_format_ctx = NULL;

    if (argc != 2) {
        printf("./streamer [file]\n");
        return (1);
    }

    if (avformat_open_input(&av_format_ctx, argv[1], NULL, NULL) != 0) {
        printf("Could not open input file.");
        return (1);
    }

    if (avformat_find_stream_info(av_format_ctx, NULL) != 0) {
        printf("Could not find stream information.");
        return (1);
    }

    decode_raw(av_format_ctx);
    avformat_close_input(&av_format_ctx);
    return (0);
}

What I tried

• Check endianness and if I correctly imported the raw file in Audacity
• Execute the corresponding ffmpeg command ffmpeg -i video.mp4 -f f32le output.raw (my code output AV_SAMPLE_FMT_FLT) to compare both files.

I hexdumped both files and found this.

// 96 1f 03 3f - 22 03 0c 3f
// Doesn't exist in the output of my program?

5581a0  7c ad 6f bc 96 1f 03 3f 4f 01 25 3e 22 03 0c 3f  |.o....?O.%>"..?   // ffmpeg
5580d0  7c ad 6f bc 4f 01 25 3e 3a d2 89 3e 7c d7 9a 3e  |.o.O.%>:..>|..>   // my implementation

---

Edit #1

After an endless succession of disappointing experiences, AAC audio streams appear to be corrupted after decoding. However, the raw PCM output from ffmpeg works well for MP4.

I tried to resample the audio frames with swr_convert but it is too poorly documented and I turned into a lot of issues.

Answer 1

Problem

After printing the information about the audio stream. I noticed than AAC (the audio codec of the mp4 file) doesn't support non planar format (packed).

// Explicitly request non planar data.
codec_ctx->request_sample_fmt = av_get_packed_sample_fmt(codec_ctx->sample_fmt);

Since the requested format wasn't supported, the audio stream of the mp4 file was decoded as planar, unlike the mp3 file.

---------
Codec: MP3 (MPEG audio layer 3)
Supported sample formats: fltp, flt        # MP3 support non planar
---------
Stream:              0
Sample Format:    fltp
Sample Rate:     48000
Sample Size:         4
Channels:            2
Planar Output:      yes

---------
Codec: AAC (Advanced Audio Coding)
Supported sample formats: fltp             # AAC doesn't support non planar
---------
Stream:              1
Sample Format:    fltp
Sample Rate:     44100
Sample Size:         4
Channels:            2
Planar Output:      yes

---

Solution

To solve the problem, I deleted the above line to keep the streams planar. I also had to change the way I wrote in the file.

As the format is planar LR, LR, LR and not packed LL LL RR RR, I had to manually write each channel alternately.

Because writing byte by byte takes a long time, I wrote a function that writes to a buffer before writing the buffer to the file.

void audio_pack_stream(AVCodecContext* codec_ctx, AVFrame *frame, uint8_t *dst, int *size)
{
    int bytes = av_get_bytes_per_sample(codec_ctx->sample_fmt);
    int actual = 0;

    for (int i = 0; i < frame->nb_samples; i++) {
        for(int j = 0; j < codec_ctx->channels; j++)
            for (int k = 0; k < bytes; k++)
                dst[*size++] = frame->extended_data[j][actual + k];
        actual += bytes;
    }
    return (size);
}

// After avcodec_receive_frame

uint8_t output[4096 * 8];
int size;

audio_pack_stream(codec_ctx, frame, output, &size);
write(fd, output, size);