Pin non-zoomable markers on a map with svg-pan-zoom

Question

I am trying to make a map that will contain markers at certain locations. The map is an svg file and the markers could be too. I am wondering what would be the best way to go to have pan and zoom available to the map svg, but also keep the markers' position relative to the svg fixed and size as well.

If I group both the marker and the map svg together and call svgPanZoom, they're grouped together so when I zoom in the map, the marker also is zoomed in (I want to keep its size constant).