Azure AppService unable to connect to SQL database

Question

I have an Azure AppService written in C# that connects to a SQL Server database hosted outside of Azure using NHibernate. The connection string looks like this:

Data Source=tcp:SQL1234.3rdpartyserver.net;MultipleActiveResultSets=true;Initial Catalog=DB_SQL1234;User Id=****;Password=****;

Most of the time everything works fine, but occasionally my AppService loses the connection, and I am getting the following exception:

 System.Data.SqlClient.SqlException: A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: TCP Provider, error: 0 - A connection
 attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.) ---> System.ComponentModel.Win32Exception: A connection
 attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond --- End of inner exception stack trace
 ---
 at System.Data.SqlClient.SqlInternalConnectionTds..ctor(DbConnectionPoolIdentity identity, SqlConnectionString connectionOptions, SqlCredential credential, Object providerInfo, String newPassword, SecureString newSecurePassword, Boolean redirectedUserInstance, SqlConnectionString userConnectionOptions, SessionData reconnectSessionData, Boolean applyTransientFaultHandling, String accessToken)
 at System.Data.SqlClient.SqlConnectionFactory.CreateConnection(DbConnectionOptions options, DbConnectionPoolKey poolKey, Object poolGroupProviderInfo, DbConnectionPool pool, DbConnection owningConnection, DbConnectionOptions userOptions)
 at System.Data.ProviderBase.DbConnectionFactory.CreatePooledConnection(DbConnectionPool pool, DbConnection owningObject, DbConnectionOptions options, DbConnectionPoolKey poolKey, DbConnectionOptions userOptions)
 at System.Data.ProviderBase.DbConnectionPool.CreateObject(DbConnection owningObject, DbConnectionOptions userOptions, DbConnectionInternal oldConnection)
 at System.Data.ProviderBase.DbConnectionPool.UserCreateRequest(DbConnection owningObject, DbConnectionOptions userOptions, DbConnectionInternal oldConnection)
 at System.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, UInt32 waitForMultipleObjectsTimeout, Boolean allowCreate, Boolean onlyOneCheckConnection, DbConnectionOptions userOptions, DbConnectionInternal& connection)
 at System.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal& connection)
 at System.Data.ProviderBase.DbConnectionFactory.TryGetConnection(DbConnection owningConnection, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal oldConnection, DbConnectionInternal& connection)
 at System.Data.ProviderBase.DbConnectionInternal.TryOpenConnectionInternal(DbConnection outerConnection, DbConnectionFactory connectionFactory, TaskCompletionSource`1 retry, DbConnectionOptions userOptions)
 at System.Data.ProviderBase.DbConnectionClosed.TryOpenConnection(DbConnection outerConnection, DbConnectionFactory connectionFactory, TaskCompletionSource`1 retry, DbConnectionOptions userOptions)
 at System.Data.SqlClient.SqlConnection.TryOpen(TaskCompletionSource`1 retry)
 at System.Data.SqlClient.SqlConnection.Open()
 at NHibernate.Connection.DriverConnectionProvider.GetConnection()
 at NHibernate.Tool.hbm2ddl.SuppliedConnectionProviderConnectionHelper.Prepare()
 at NHibernate.Tool.hbm2ddl.SchemaMetadataUpdater.GetReservedWords(Dialect dialect, IConnectionHelper connectionHelper)
 at NHibernate.Tool.hbm2ddl.SchemaMetadataUpdater.Update(ISessionFactoryImplementor sessionFactory)
 at NHibernate.Impl.SessionFactoryImpl..ctor(Configuration cfg, IMapping mapping, Settings settings, EventListeners listeners)
 at NHibernate.Cfg.Configuration.BuildSessionFactory()
 at FluentNHibernate.Cfg.FluentConfiguration.BuildSessionFactory() --- End of inner exception stack trace

This starts happening out of nowhere: I am not updating any connection string, not restarting my AppService, etc. The application only fails to connect to the database from Azure. If I launch the application locally, everything works as expected using the same connection string. Additionally, I can connect to the DB fine from SSMS.

Sometimes restarting my AppService helps, and the connectivity is restored after a restart. But sometimes it doesn't help.

I am suspecting the connection may be blocked by Azure's firewall, but I don't know how to check this. My application is using a B1 App Service plan, and I haven't created any custom firewalls, or load balancers in my Azure Portal. In fact, this AppService is the only resource that I currently have.

Any ideas what might be causing this, and hw to fix it?

Answer 1

You're most likely hitting SNAT exhaustion. Under Diagnose and Solve Problems Blade search for "TCP Connections" which will show you how many TCP connections your application is making. If there is a high number of connections to SQL (~128+) you're application is in a state that will most likely run into timeout exceptions.

App Services run in the 201-400 range for the multi-tenant app services so once you application makes 128 individual TCP connections to a specific destination IP/port you'll likely see these issues. https://learn.microsoft.com/en-us/azure/load-balancer/load-balancer-outbound-connections

My recommendations would be in the following order:

1. Make sure to use connection pooling to limit the individual number of tcp connections. I've worked with customers who had 1000s of tcp connections and after using connection pooling for all their connections it dropped down to sub 100. The plan size does not make a difference for this particular issue. https://learn.microsoft.com/en-us/dotnet/framework/data/adonet/sql-server-connection-pooling

2. Use Regional VNET integration - SNAT ports do not come into play with VNET integration. You can then utilize service endpoints to route traffic to Azure SQL. https://learn.microsoft.com/en-us/azure/app-service/web-sites-integrate-with-vnet#regional-vnet-integration

3. Scale out the application to multiple instances - This helps spread the requests and outbound SQL connection across multiple VMs

4. Use an ASE - This is a much more expensive option but just wanted to add it for answer completeness sake. The SNAT ports depend on the number of instances you have as seen in the doc above

Answer 2

In your SQL Server firewall you can configure/allow the outbound IP addresses for your App Service. You can get these IPs from property section of your App Service or by using CLI.
Inbound and outbound IP addresses in Azure App Service

Answer 3

This is a very common and frequent problem and it happens due network instability. The way to solve is just wrap the code block using a retry pattern.

https://learn.microsoft.com/en-us/azure/architecture/patterns/retry