Running Mellanox libvma as non-root user, how to work with the missing LD_LIBRARY_PATH after setcap

Question

I am trying to run my project with Mellaneox kernel-bypass library libvma.

After following the user guide to setup the necessary permission with setuid and setcap, LD linker is running in a "secure-execution" mode which ignores the LD_LIBRARY_PATH completely. However, my project is in a structure where the executable has to link with several inhouse-built library, e.g.
my_program => libmy-core.so , libmy-network.so, libmy-util.so

Then, without the LD_LIBRARY_PATH, it ends up with ./my_program: error while loading shared libraries: ....

I tried the way to set the rpath during compilation to avoid runtime LD searching, but it doesn't work perfectly because the built .so will be deployed to different account without a consistent path. (Not perfectly, because I can still hack it by adding a relative path "." in the rpath)

I wonder will there be any better way to work-around this security protection?

For reference, libvma requires at least the following to work:

1. sudo chmod u+s /usr/lib64/libvma*
2. sudo setcap cap_net_raw,cap_net_admin+ep my_program
3. LD_PRELOAD=libvma.so my_program 

Also tried to create a wrapper script (as copilot suggested), but sounds like the child process can't inherit the capabilities set in the parent.sh, even the cap is granted to all +epi group.