OpenCV stereoCalibrate always returns totally different results

Question

I want to experiment with stereo imaging and therefore, installed two webcams which are aligned nearly parallel next to each other.

I detect corners on a charuco chessboard and calculate the fundamental matrix using stereoCalibrate from OpenCV. After that, I am calculating epiploar lines of the found pixel coordinates of the first image pair and visualize them using the code from the OpenCV tutorial.

The whole code is available here.

I ran this program multiple times, but what confuses me is that the fundamental matrix is everytime totally different.

One time, it is:

0.0 -0.00002 0.00411
0.00002 0.00000 -0.02593
-0.00624 0.02016 1

And another time, it is:

-0.00001 0.00004 0.01019
-0.00004 -0.00001 0.0887
0.00018 -0.08702 1

Further, one time the epipoles are within the image plane:

and another run, they are completely outside:

And each line should cross a dot since a line is an epiline and the dot is a found pixel in the other image. But this is also not the case.

As far as I understand, the epipoles should never change if the cameras are not moved, right? (which I don't do) Of course, for each run I use different images from my cameras so I understand that there should by a slight difference. But at least the epipoles should be at quite the same position. So do you have an idea why the fundamental matrix changes so drastically during runs?