How to raycast from mouse position in scene view?

Question

I'm trying to raycast in scene view based mouse position. I pretty much want to mimic the unity's behavior when you click some where it selects that objects except I wish to get all the objects through that ray.

Here's exactly what I want to do:

1. Press a button on the inspector of a script to start raycasting.
2. As user moving through scene it selects (keeps record of) all the objects that intersect with ray from current mouse position.
3. Then user clicks in the scene view to stop the raycasting and the last set of selected objects is shown in the inspector.

I've search and not seen a solution that works. It seems like such a trivial thing to be able to do.

So if anyone can create a simple script of function that works that would be great.

Here's a link to what I have so far: https://gist.github.com/lordlycastle/fdc919da37585410309df3231ed03e00

The problem isn't that I can't raycast through all. You can do that with RaycastAll. The problem is getting the correct Ray (ie origin and direction) from scene camera based on mouse position.

Answer 1

Unity provides a utility function called HandleUtility.GUIPointToWorldRay this should provide accurate rays in the SceneGUI function.

private void OnSceneGUI()
{

    Event e = Event.current;

    // check mouse down event
    if (e.type != EventType.MouseDown)
    {
        return;
    }

    // check left mouse button
    if (e.button != 0)
    {
        return;
    }
    
    // create OnSceneGUI ray
    Ray ray = HandleUtility.GUIPointToWorldRay(Event.current.mousePosition);

    // check hit
    if (!Physics.Raycast(ray, out RaycastHit hit))
    {
        return;
    }

    //
    // Use ray cast hit here
    //


    // tell event to no longer propergate
    e.Use();

}

Answer 2

After many searches I was able to create the behaviour I was trying to create. Hopefully it'll help someone. If you want to be able to raycast with an custom editor window then you should use @Draco18s 's answer. If you want to write that code in a MB then this will work.

    // first add your function to be called on redraw
    SceneView.onSceneGUIDelegate += UpdateRaycast;

    // then in that function raycast
    Vector3 mousePosition = Event.current.mousePosition;
    mousePosition.y = SceneView.currentDrawingSceneView.camera.pixelHeight - mousePosition.y;
    mousePosition = SceneView.currentDrawingSceneView.camera.ScreenToWorldPoint(mousePosition);
    mousePosition.y = -mousePosition.y;

    mouseRay = HandleUtility.GUIPointToWorldRay(Event.current.mousePosition);
    raycastHits = Physics.RaycastAll(mouseRay, maxDistance)
    // be sure to remove the event if you're not using it or call it from an update function
    SceneView.onSceneGUIDelegate -= UpdateRaycast;

See whole script if you wish here.

Answer 3

Oof, getting an accurate ray from the mouse in the scene view is annoyingly difficult, because the raw mouse position (Event.current.mousePosition) is relative to the entire Unity application window, not relative to the scene viewport.

Its a good thing I'd had some code laying around from a thing I did about a year ago.

private void OnGUI() {
    //do stuff

    //dump out if things are null or otherwise in a problematic state
    if(SceneView.lastActiveSceneView.camera == null || SceneView.lastActiveSceneView != mouseOverWindow && Event.current.type == EventType.Repaint) return;

    Vector3 mousePosition = Event.current.mousePosition;

    mousePosition.x += position.x; //add the position of our editor gui window
    mousePosition.y += position.y;

    mousePosition.x -= SceneView.lastActiveSceneView.position.x; //subtract off the position of the scene view window
    mousePosition.y -= SceneView.lastActiveSceneView.position.y;
    mousePosition.x /= SceneView.lastActiveSceneView.position.width; //scale to size
    mousePosition.y /= SceneView.lastActiveSceneView.position.height;
    mousePosition.z = 1; //set Z to a sensible non-zero value so the raycast goes in the right direction
    mousePosition.y = 1 - mousePosition.y; //invert Y because UIs are top-down and cameras are bottom-up

    Ray ray = SceneView.lastActiveSceneView.camera.ViewportPointToRay(mousePosition);

    //do your raycast
}

I had all this in a function called from OnGui, as my editor window did other stuff, only one part of which involved the user selecting verticies of a mesh, but I'd print various messages like GUILayout.Label("Activate a scene view!"); if the last active scene view was null, which would happen some times. I think it was because if the focused window/tab wasn't a scene view tab, it would be null, so the user would have to activate one, de-focusing the editor window, but OnGui still gets called, and calling Focus() would re-focus the window, but we'd retain a valid scene view.

I haven't tried this on an ExecuteInEditMode MonoBehaviour script. Certainly some of the manipulation of the mouse position won't need to be done, but others might. You're going to need to log the raw mouse position as you move your mouse around and see what you get, then do what needs to be done to make it bounded by [0,1].