Why am I seeing poor encoding performance on AWS with NV_ENC for first run of app

Question

I have cross posted on repost.aws so apologies to those who are reading this twice.

I have a windows app that interactively streams content from a directx 3D rendering over webrtc, after encoding with FFMPEG using h264_nvenc encoder, support for which I've compiled in.

The app currently starts running 2 minutes after the instance starts. It is started via a .bat file that's launched on startup, and has a timeout call in the .bat file to achieve the delay.

I'm using a g5.2xlarge, so the GPU is an A10G. I consistently get low encoding frame rates (under 10FPS) on any instance that starts up like this.

However, if I then reboot the application, encoding rates go up to the high 20s FPS.

As far as my app is concerned, it is finding the encoder ok both times. I have verified with nvidia-smi encodersessions that the encoding rate is 9FPS on first run and then jumps to around 29-30 on the second.

In all cases, nvidia-smi shows the app is using the GPU.

When using a different instance type - g4dn.2xlarge - the encoding rate is as expected from the first run of the app. This uses a T4.

I thought I had solved the problem by waiting (with a loop in the startup script that tests every 5s) until nvidia-smi returned successfully before starting my app, because if I logged into it early enough I would get a "9" return code, indicating that the driver hadn't loaded yet.

Interestingly the logging indicates that when I'm not logged into the instance (via fleet manager remote desktop), nvidia-smi returns 0 the first time it is called.

Any thoughts on what could be causing this - seems like a driver issue - and any workarounds?

EDIT:

I've since even modified my app so that with a command line option it will just encode a requested number of generated frames, and then exit. I run it with that flag four times before starting it without the flag, and I still get the same poor encoding when it starts up proper.

I've removed some extraneous detail from the post as well.