vtkXOpenGLRenderWindow error in a terminal in VNC

Question

I run a vnc server on my workstation and connect with another computer. Both of the server and client are running Debian Jessie:

$ uname -a
Linux debian-VAIO 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt25-2+deb8u2 (2016-06-25) x86_64 GNU/Linux

Xfce is installed for the vnc server and xstartup is:

$ cat ~/.vnc/xstartup
#!/bin/sh
xrdb $HOME/.Xresources
xsetroot -solid grey
export XKL_XMODMAP_DISABLE=1
exec startxfce4

In the ~/.bashrc, I have added the lines below:

export LIBGL_ALWAYS_INDIRECT=y
export LD_PRELOAD='/usr/lib/x86_64-linux-gnu/libstdc++.so.6'

Currently, when I ssh to the server with ssh -X, the $DISPLAY is localhost:10.0, and the vtk related command will launch an X window. However, if a vnc server is launched in the ssh terminal, logging into xfce and opening a terminal in it, the $DISPLAY is :1.0 and vtk fails to launch an X window, with the following message:

Error: In /home/orobix/Desktop/vmtk-build/VTK/Rendering/OpenGL/vtkXOpenGLRenderWindow.cxx, line 394
vtkXOpenGLRenderWindow (0x2c30f10): Could not find a decent visual

Error: In /home/orobix/Desktop/vmtk-build/VTK/Rendering/OpenGL/vtkXOpenGLRenderWindow.cxx, line 394
vtkXOpenGLRenderWindow (0x2c30f10): Could not find a decent visual

Error: In /home/orobix/Desktop/vmtk-build/VTK/Rendering/OpenGL/vtkXOpenGLRenderWindow.cxx, line 394
vtkXOpenGLRenderWindow (0x2c30f10): Could not find a decent visual

Error: In /home/orobix/Desktop/vmtk-build/VTK/Rendering/OpenGL/vtkXOpenGLRenderWindow.cxx, line 613
vtkXOpenGLRenderWindow (0x2c30f10): GLX not found.  Aborting.

I think it might be relate to the missed config of X server in xstartup of vnc server. But I don't know how to do it. Could anyone help me debug it? Any further information will be provided if needed. Thanks!

20160823 Update

I accepted the suggestion of VirtualGL + TurboVNC and installed the two components. A simple config using vglserver_config was done according to http://www.virtualgl.org/vgldoc/2_1_1/#hd009001 part 6.1. Then I made vncserver of TurboVNC to run xfce with xstartup.turbovnc:

#!/bin/sh
unset SESSION_MANAGER
unset DBUS_SESSION_BUS_ADDRESS
OS=`uname -s`
exec startxfce4

On the client, I use vncviewer of TurboVNC to connect the server. The desktop looks quite different from that in a default vncserver. Then I tried

/opt/VirtualGL/bin/vglrun vmtkimageviewer -ifile image_volume_voi.vti

in which the vmtkimageviewer should open up a window, but finally it gives out an error:

Executing vmtkimageviewer ...
X Error of failed request:  GLXBadContext
  Major opcode of failed request:  156 (GLX)
  Minor opcode of failed request:  6 (X_GLXIsDirect)
  Serial number of failed request:  17
  Current serial number in output stream:  16

And unfortunately, I find that with ssh -X, I cannot launch the X window as I did before, although $DISPLAY is still localhost:10.0. The output is:

Executing vmtkimageviewer ...
X Error of failed request:  BadValue (integer parameter out of range for operation)
  Major opcode of failed request:  156 (GLX)
  Minor opcode of failed request:  3 (X_GLXCreateContext)
  Value in failed request:  0x0
  Serial number of failed request:  37
  Current serial number in output stream:  38

But I think I am closer to the fact, as it looks like VirtualGL works(?). What to do for the next?

Answer 1

Finally, I got everything done. VirtualGL only is enough for my purpose. Just install and config as the section 6.2 & 6.3 in its user's guide. Use the default vncserver and vncviewer as usual, TurboVNC is not needed. Use vglrun when you need an OpenGL support. One modification is that I should remove export LIBGL_ALWAYS_INDIRECT=y in my .bashrc.

It runs fluent in my case. @datenwolf: Thank you again!

Answer 2

Plain and simple the X server variant that's being used by Xrdp (either Xvnc or Xvfb) simply doesn't support OpenGL / GLX and thus programs that need OpenGL will not work in that configuration.

Fallback to Mesa swrast or llvmpipe is possible though: http://www.mesa3d.org/llvmpipe.html