FFmpeg - resampled audio with much noise

Question

I'm not familiar with auido resampling. I tried to resample auido streams from two videos. The first one's output was close to the original but with noise, the other one was almost full of noise.

Information for the first one

128 kb/s, 48.0kHz, 2 channels, AACLC

Information for the second one

384 kb/s, 48.0 kHz, 6channels, AACLC

I found that, when I set the sample size 16, the frist one worked quit good but still with noise. The other one worked too bad but still had sound. What and how to determine the output sample size? Although I used channels * av_get_bytes_per_sample((AVSampleFormat)output_fmt) as the output sample size because I wanted it to be the same as the original, it had no sound at all.

MyResampling.cpp

bool MyResample::open(AVCodecParameters* par) {
    if (!par) {
        std::cout << "par is null" << std::endl;
        return false;
    }   
    audio_context = swr_alloc_set_opts(
        audio_context, av_get_default_channel_layout(2), (AVSampleFormat)output_fmt,
        par->sample_rate, av_get_default_channel_layout(par->channels), (AVSampleFormat)par->format, par->sample_rate,
        0, 0);

    avcodec_parameters_free(&par);
    int ret = swr_init(audio_context);
    if (ret != 0) {
        std::cout << "failed to open audio codec" << std::endl;
    }
    return true;
}

int MyResample::resample(AVFrame* frame, unsigned char* output)
{
    if (!frame)
        return 0;
    if (!output)
        av_frame_free(&frame);

    uint8_t* data[2] = { 0 };
    data[0] = output;
    int ret = swr_convert(audio_context, data, frame->nb_samples, (const uint8_t**)frame->data, frame->nb_samples);
    //int size =  ret * frame->channels * av_get_bytes_per_sample((AVSampleFormat)output_fmt);
    int size = av_samples_get_buffer_size(nullptr, frame->channels, frame->nb_samples, (AVSampleFormat)output_fmt, 1);
    if (ret < 0)
        return ret;
    return size;
}

MyAudioPlayer.cpp

bool open()
{
    close();
    QAudioFormat fmt;
    fmt.setSampleRate(sample_rate); // from audioStream->codecpar->sample_rate
    fmt.setSampleSize(16); //
    fmt.setChannelCount(channels); // from audioStream->codecpar->channels
    fmt.setCodec("audio/pcm");
    fmt.setByteOrder(QAudioFormat::LittleEndian);
    fmt.setSampleType(QAudioFormat::UnSignedInt);
    output = new QAudioOutput(fmt);
    io = output->start();
    if (io)
        return true;
    return false;
}

bool write(const unsigned char* data, int data_size)
{
    if (!data || data_size <= 0)
        return false;
    if (!output || !io)
    {
        return false;
    }
    int size = io->write((char*)data, data_size);
    if (data_size != size)
        return false;
    return true;
}

main.cpp

MyAudioPlayer::open();
unsigned char* pcm = new unsigned char[1024 * 1024];
if (demux.get_media_type() == 1) { // audio 
    audio_decode.sendPacket(pkt);
    AVFrame* frame = audio_decode.receiveFrame();
    int len = resample.resample(frame, pcm);
    while (len > 0) {
        if (MyAudioPlayer::check_space() >= len) {
            MyAudioPlayer::write(pcm, len);
            break;
        }
        msleep(1);
    }               
}

Answer 1

If you have troubles with the final quality and noise probably you are misunderstanding the proper way to perform a resampling or there is a bug in your configuration.

Take a look into this example: libswresample-example.

I am not familiar with the FFmpeg API because to do resampling I tend to use libsamplerate.

Regarding old example, those are the steps to perform a basic resample with FFMPEG:

1. Start by configuring your resampling context:

//Set up resampling context
SwrContext *swr = swr_alloc();
av_opt_set_channel_layout(swr, "in_channel_layout", AV_CH_LAYOUT_STEREO, 0);
av_opt_set_channel_layout(swr, "out_channel_layout", AV_CH_LAYOUT_STEREO, 0);
av_opt_set_int(swr, "in_sample_rate", 44100, 0);
av_opt_set_int(swr, "out_sample_rate", 22050, 0);
av_opt_set_sample_fmt(swr, "in_sample_fmt", AV_SAMPLE_FMT_FLT, 0);
av_opt_set_sample_fmt(swr, "out_sample_fmt", AV_SAMPLE_FMT_FLT, 0);
swr_init(swr);

Depending on your input data types and the format you expect as an output, you will need to specify the right format. This is the equivalence in C++ standard:

----------------------------------------
| *AV_SAMPLE_FMT_S16* | `std::int16_t` |
| *AV_SAMPLE_FMT_S32* | `std::int32_t` |
| *AV_SAMPLE_FMT_FLT* | `float`        |
| *AV_SAMPLE_FMT_DBL  | `double`       |
| *AV_SAMPLE_FMT_U8P* | `std::uint8_t` |
| ...                 |                |

2. Get your data from whatever place in the right format and estimate your sampling count.

After that, you can perform the resampling in few steps:

3. Estimate the number of output samples

uint8_t* out_samples;
int out_num_samples = av_rescale_rnd(swr_get_delay(swr, in_samplerate) + in_num_samples, out_samplerate, in_samplerate, AV_ROUND_UP);

4. Allocate the memory for the output file

av_samples_alloc(&out_samples, NULL, out_num_channels, out_num_samples, AV_SAMPLE_FMT_FLT, 0);

5. Convert the input data into the expected output format

out_num_samples = swr_convert(swr, &out_samples, out_num_samples, &in_samples, in_num_samples);

6. Do not forget to free your memory

av_freep(&out_samples);
swr_free(&swr);

If you have noise, probably the input formats and output formats are not the proper one or the resampling quality is low.

For instance, do not panic if you get fewer samples than what you expected. It is the common behavior because of the way the filtering works. To get the remaining trailing you can perform the step 5 with NULL as input, which will flush the internal data.