CODEWITHSUNDEEP
cffmpeg
loneraver
loneraver

Reputation: 1422

Convert ffmpeg frame into array of YUV pixels in C

I'm using the ffmpeg C libraries and trying to convert an AVFrame into a 2d array of pixels with YUV* components for analysis. I figured out how to convert the Y component for each pixel.:

uint8_t y_val = pFrame->data[0][pFrame->linesize[0] * y + x];

Since all frames have a Y component this is easy. However most digital video do not have a 4:4:4 chroma subsampling, so getting the UV components is stumping me.

I'm using straight C for this project. No C++. An ideas?

*Note: Yes, I know it's technically YCbCr and not YUV.

Edit:

I'm rather new to C so it might not be the prettiest code out there.

When I try: 

VisYUVFrame *VisCreateYUVFrame(const AVFrame *pFrame){
    VisYUVFrame *tmp = (VisYUVFrame*)malloc(sizeof(VisYUVFrame));
    if(tmp == NULL){ return NULL;}
    tmp->height = pFrame->height;
    tmp->width = pFrame->width;

    tmp->data = (PixelYUV***)malloc(sizeof(PixelYUV**) * pFrame->height);
    if(tmp->data == NULL) { return NULL;};

    for(int y = 0; y < pFrame->height; y++){
        tmp->data[y] = (PixelYUV**)malloc(sizeof(PixelYUV*) * pFrame->width);
        if(tmp->data[y] == NULL) { return NULL;}

        for(int x = 0; x < pFrame->width; x++){
            tmp->data[y][x] = (PixelYUV*)malloc(sizeof(PixelYUV*));
            if(tmp->data[y][x] == NULL){ return NULL;};
            tmp->data[y][x]->Y = pFrame->data[0][pFrame->linesize[0] * y + x];
            tmp->data[y][x]->U = pFrame->data[1][pFrame->linesize[1] * y + x];
            tmp->data[y][x]->V = pFrame->data[2][pFrame->linesize[2] * y + x];

        }
    }

    return tmp;

Luma works but when I run Valgrind, I get

0x26 1 InvalidRead Invalid read of size 1 0x100003699 /Users/hborcher/Library/Caches/CLion2016.2/cmake/generated/borcherscope-8e83e7dd/8e83e7dd/Debug/VisCreator2 VisCreateYUVFrame /Users/hborcher/ClionProjects/borcherscope/lib visualization.c 145 0x100006B5B /Users/hborcher/Library/Caches/CLion2016.2/cmake/generated/borcherscope-8e83e7dd/8e83e7dd/Debug/VisCreator2 render /Users/hborcher/ClionProjects/borcherscope/lib/decoder simpleDecoder2.c 253 0x100002D24 /Users/hborcher/Library/Caches/CLion2016.2/cmake/generated/borcherscope-8e83e7dd/8e83e7dd/Debug/VisCreator2 main /Users/hborcher/ClionProjects/borcherscope/src createvisual2.c 93 Address 0x10e9f91ef is 0 bytes after a block of size 92,207 alloc'd 0x100013EEA /usr/local/Cellar/valgrind/3.11.0/lib/valgrind/vgpreload_memcheck-amd64-darwin.so malloc_zone_memalign 0x1084B5416 /usr/lib/system/libsystem_malloc.dylib posix_memalign 0x10135D317 /usr/local/Cellar/ffmpeg/3.0.2/lib/libavutil.55.17.103.dylib av_malloc

0x27 1 InvalidRead Invalid read of size 1 0x1000036BA /Users/hborcher/Library/Caches/CLion2016.2/cmake/generated/borcherscope-8e83e7dd/8e83e7dd/Debug/VisCreator2 VisCreateYUVFrame /Users/hborcher/ClionProjects/borcherscope/lib visualization.c 147 0x100006B5B /Users/hborcher/Library/Caches/CLion2016.2/cmake/generated/borcherscope-8e83e7dd/8e83e7dd/Debug/VisCreator2 render /Users/hborcher/ClionProjects/borcherscope/lib/decoder simpleDecoder2.c 253 0x100002D24 /Users/hborcher/Library/Caches/CLion2016.2/cmake/generated/borcherscope-8e83e7dd/8e83e7dd/Debug/VisCreator2 main /Users/hborcher/ClionProjects/borcherscope/src createvisual2.c 93 Address 0x10e9f91ef is 0 bytes after a block of size 92,207 alloc'd 0x100013EEA /usr/local/Cellar/valgrind/3.11.0/lib/valgrind/vgpreload_memcheck-amd64-darwin.so malloc_zone_memalign 0x1084B5416 /usr/lib/system/libsystem_malloc.dylib posix_memalign 0x10135D317 /usr/local/Cellar/ffmpeg/3.0.2/lib/libavutil.55.17.103.dylib av_malloc

Upvotes: 0

Views: 968

Answers (1)

Ronald S. Bultje
Ronald S. Bultje

Reputation: 11184

If you are OK with hardcoding the chroma subsampling, e.g. you know the frame data format is 4:2:0, it's simple:

int uvy = y >> 1, uvx = x >> 1;
uint8_t u_val = pFrame->data[1][pFrame->linesize[1] * uvy + uvx];
uint8_t v_val = pFrame->data[2][pFrame->linesize[2] * uvy + uvx];

If you want it to be more generic, use this:

AVPixFmtDescriptor *desc = av_pix_fmt_desc_get(pFrame->format);
int uvy = y >> desc->log2_chroma_h, uvx = x >> desc->log2_chroma_w;
uint8_t u_val = pFrame->data[1][pFrame->linesize[1] * uvy + uvx];
uint8_t v_val = pFrame->data[2][pFrame->linesize[2] * uvy + uvx];

That should work for all cases to get pixels at any x,y location. However, don't use this to convert buffers of any chroma subsampling into 4:4:4 arrays, it will have visual artifacts. For display on-screen, use the original data and use per-plane sizing in e.g. your openGL shaders to convert the original arrays on-screen to the desired target resolution. To convert to 4:4:4 for other use cases, use libswscale.

Upvotes: 1

Related Questions