Is there a way to enable `module-suspend-on-idle` only for specific sinks

Question

I have an HDMI sound card device that is extremely slow to exit suspend. The primary problem this creates is when I'm not playing any audio on my machine, any brief audio alerts wakeup the card, but are not produced by the speaker. Any subsequent alerts for a short period will work fine, but after a brief period of quiet PulseAudio re-suspends the card.

As such, I'd like to avoid using the PulseAudio module-suspend-on-idle module specifically for this card, but I'd like to know if there is a way to configure it to remain enabled for any "regular" sound cards on my system.

Currently my solution (on Ubuntu 18.04) has been to duplicate /etc/pulse/defaults.pa to ~/.config/pulse/defaults.pa and just comment out the load-module line, but I'm hoping for something a bit more targeted.