Applying gabor filter independant of frontier colors

Question

I'm a set of Gabor filters to an image in order to get the vanishing point. I'm applying four filters with an angle of (0°, 45°, 90° and 135°). I then take the most relevant pixel and compute the angle for this particular pixel depending on the score of the four filters. I then give a positive score to every pixels that is on the way, with a wide angle of 15°. I get some good results when I'm trying with an image that is white with black lines.

However, if I try with a reversed image, white lines on a black background the angles given are 90° shifted.

Is there a parameter in the Gabor filter that allows to be more independent of the color of the edges? Or a technique to take this into account?