NVEnc encoding and MediaCodec decoding producing corrupted frames

Question

I'm capturing frames using Desktop Duplication API

hr = pDup->AcquireNextFrame(wait, &frameInfo, &pResource);

preprocessing them (RGB to NV12)

then encoding them using the default NVEnc present for HEVC

pEnc->CreateDefaultEncoderParams(&encInitParams, NV_ENC_CODEC_HEVC_GUID, NV_ENC_PRESET_LOW_LATENCY_DEFAULT_GUID);
pEnc->CreateEncoder(&encInitParams);

Up until here everything works, saving the output into file shows that everything works perfectly. Sending them over network to Android is a different story. On Android I'm using MediaCodec, I get the first frame from the decoder and use it as csd-0 to configure the async decoder with the correct profile. On my callbacks I have this:

mCodec.setCallback(new MediaCodec.Callback() {
        @Override
        public void onInputBufferAvailable(@NonNull MediaCodec mediaCodec, int i) {
            if(data!=null){
                ByteBuffer inputBuffer = mediaCodec.getInputBuffer(i);
                inputBuffer.put(data, 0, data.length);

                mediaCodec.queueInputBuffer(i, 0, data.length, 0, 0);

            }
        }

        @Override
        public void onOutputBufferAvailable(@NonNull MediaCodec mediaCodec, int i, @NonNull MediaCodec.BufferInfo bufferInfo) {

            if(data!=null) {
                mediaCodec.releaseOutputBuffer(i, true);
            }
        }

data is being set by an onReceiveMessage(byte[]). The results that I get are frames that don't fit with each other and the result is just a smudged mess (see below for a screenshot). The only way it works is if I force the NVEnc to only send IDR Frames, and that's heavy on the network. Any ideas on what I'm doing wrong?