Connection to Availability Group Listener gives a 'Transport Level' Error in SSMS when I shut down Primary

Question

I have a Node1\INST1 SQL Server instance which is the Primary and a Node2\Inst2 which is the Secondary in the High Availability Group. Both shows 'Synchronized' as the status, and both show 'No Data Loss'. I have also configured the Failover correctly.

Both are up and I have my SSMS open with a connection to the Listener,1496. I fire a query against this one test table and it returns the correct results.

Next, I stop the SQL Service on the Primary (to test the Failover). The 'Automatic Failover' happens correctly and I can see now the Node2\Inst2 is Primary.

But my query on the listener connection window gives that typical:

A transport level occurred

error if I fire that query again. Shouldn't that query work seamlessly? Like tomorrow, If I have a .NET app with the connection string pointing to the listener, what kind of issues will I face when the Primary goes down?

Please let me know If I am testing it wrong or if I need to do something else.

Thanks