"InvalidOperationException: StatusCode cannot be set because the response has already started" error from App Services to AI Search (private network)

Question

Components

• Azure App Services
• Azure AI Search

Overview

We are deploying the sample-app-aoai-chatGPT application through Azure AI Studio. Everything works while over public networks, but when we try to create a private network between Azure App Services to Azure AI Search, the application fails with the following:

Invalid AzureCognitiveSearch configuration detected: Call to get ACS index failed. Check you are using correct index, instance and api_key.

Private networking details

• Azure App Services is injected into a subnet (virtual network-integrated)
• Azure AI Search has a private endpoint on a different subnet in the same virtual network
• Azure AI Search has public access disabled

Troubleshooting

• Via Kudu and nslookup, confirmed that DNS is resolving to the private IP address
• If Azure AI Search's network has public access enabled, everything works
• REST API calls to the Azure AI Search private endpoint returns expected results
• Outbound access from Azure App Services to a private endpoint on Azure AI services works

Diagram

Example error

App Services logs

2024-02-05T16:18:53.465043284Z: [ERROR]  [2024-02-05 16:18:53 +0000] [79] [ERROR] Error handling request
2024-02-05T16:18:53.465111184Z: [ERROR]  Traceback (most recent call last):
2024-02-05T16:18:53.465117884Z: [ERROR]    File "/tmp/8dc1e90f6f97d9f/app.py", line 398, in stream_with_data
2024-02-05T16:18:53.465122784Z: [ERROR]      response["model"] = lineJson["model"]
2024-02-05T16:18:53.465126884Z: [ERROR]                          ~~~~~~~~^^^^^^^^^
2024-02-05T16:18:53.465130584Z: [ERROR]  KeyError: 'model'
2024-02-05T16:18:53.465134384Z: [ERROR]  
2024-02-05T16:18:53.465152884Z: [ERROR]  During handling of the above exception, another exception occurred:
2024-02-05T16:18:53.465157184Z: [ERROR]  
2024-02-05T16:18:53.465160484Z: [ERROR]  Traceback (most recent call last):
2024-02-05T16:18:53.465164184Z: [ERROR]    File "/opt/python/3.11.4/lib/python3.11/site-packages/gunicorn/workers/sync.py", line 184, in handle_request
2024-02-05T16:18:53.465167984Z: [ERROR]      for item in respiter:
2024-02-05T16:18:53.465171484Z: [ERROR]    File "/tmp/8dc1e90f6f97d9f/antenv/lib/python3.11/site-packages/werkzeug/wsgi.py", line 256, in __next__
2024-02-05T16:18:53.465175184Z: [ERROR]      return self._next()
2024-02-05T16:18:53.465178684Z: [ERROR]             ^^^^^^^^^^^^
2024-02-05T16:18:53.465182184Z: [ERROR]    File "/tmp/8dc1e90f6f97d9f/antenv/lib/python3.11/site-packages/werkzeug/wrappers/response.py", line 32, in _iter_encoded
2024-02-05T16:18:53.465185984Z: [ERROR]      for item in iterable:
2024-02-05T16:18:53.465189484Z: [ERROR]    File "/tmp/8dc1e90f6f97d9f/app.py", line 425, in stream_with_data
2024-02-05T16:18:53.465193184Z: [ERROR]      yield format_as_ndjson({"error" + str(e)})
2024-02-05T16:18:53.465196884Z: [ERROR]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-02-05T16:18:53.465200384Z: [ERROR]    File "/tmp/8dc1e90f6f97d9f/app.py", line 170, in format_as_ndjson
2024-02-05T16:18:53.465203984Z: [ERROR]      return json.dumps(obj, ensure_ascii=False) + "\n"
2024-02-05T16:18:53.465207684Z: [ERROR]             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-02-05T16:18:53.465211184Z: [ERROR]    File "/opt/python/3.11.4/lib/python3.11/json/__init__.py", line 238, in dumps
2024-02-05T16:18:53.465214984Z: [ERROR]      **kw).encode(obj)
2024-02-05T16:18:53.465218384Z: [ERROR]            ^^^^^^^^^^^
2024-02-05T16:18:53.465221884Z: [ERROR]    File "/opt/python/3.11.4/lib/python3.11/json/encoder.py", line 200, in encode
2024-02-05T16:18:53.465225484Z: [ERROR]      chunks = self.iterencode(o, _one_shot=True)
2024-02-05T16:18:53.465229084Z: [ERROR]               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-02-05T16:18:53.465232484Z: [ERROR]    File "/opt/python/3.11.4/lib/python3.11/json/encoder.py", line 258, in iterencode
2024-02-05T16:18:53.465236184Z: [ERROR]      return _iterencode(o, 0)
2024-02-05T16:18:53.465239685Z: [ERROR]             ^^^^^^^^^^^^^^^^^
2024-02-05T16:18:53.465243085Z: [ERROR]    File "/opt/python/3.11.4/lib/python3.11/json/encoder.py", line 180, in default
2024-02-05T16:18:53.465246785Z: [ERROR]      raise TypeError(f'Object of type {o.__class__.__name__} '
2024-02-05T16:18:53.465250285Z: [ERROR]  TypeError: Object of type set is not JSON serializable
2024-02-05T16:18:53.496122015Z: [INFO]  fail: Middleware[0]
2024-02-05T16:18:53.496560416Z: [INFO]        Failed to forward request to http://169.254.129.3:8000. Encountered a System.IO.IOException exception after 32272.399ms with message: The response ended prematurely.. Check application logs to verify the application is properly handling HTTP traffic.
2024-02-05T16:18:53.542252109Z: [INFO]  fail: Microsoft.AspNetCore.Server.Kestrel[13]
2024-02-05T16:18:53.542295609Z: [INFO]        Connection id "0HN1554QM4OJH", Request id "0HN1554QM4OJH:00000002": An unhandled exception was thrown by the application.
2024-02-05T16:18:53.545071321Z: [INFO]        System.InvalidOperationException: StatusCode cannot be set because the response has already started.
2024-02-05T16:18:53.545093721Z: [INFO]           at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ThrowResponseAlreadyStartedException(String value)
2024-02-05T16:18:53.545213221Z: [INFO]           at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.Microsoft.AspNetCore.Http.Features.IHttpResponseFeature.set_StatusCode(Int32 value)
2024-02-05T16:18:53.545217821Z: [INFO]           at Microsoft.Azure.AppService.Middleware.Forwarding.RequestForwarder.OnRequest(HttpContext context) in /__w/1/s/src/EasyAuth/Middleware.Forwarding/RequestForwarder.cs:line 90
2024-02-05T16:18:53.545238621Z: [INFO]           at Microsoft.Azure.AppService.Middleware.NetCore.AppServiceMiddleware.InvokeAsync(HttpContext context) in /__w/1/s/src/EasyAuth/Microsoft.Azure.AppService.Middleware.NetCore/AppServiceMiddleware.cs:line 140
2024-02-05T16:18:53.545834924Z: [INFO]           at Microsoft.Azure.AppService.MiddlewareShim.AutoHealing.AutoHealingMiddleware.Invoke(HttpContext context) in /__w/1/s/src/EasyAuth/Middleware.Host/AutoHealing/AutoHealingMiddleware.cs:line 59
2024-02-05T16:18:53.545862024Z: [INFO]           at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequests[TContext](IHttpApplication`1 application)

Answer 1

A special private endpoint needs to be provisioned by Microsoft. Details in these pages:

• https://learn.microsoft.com/en-us/azure/ai-services/openai/how-to/use-your-data-securely#disable-public-network-access-1
• https://github.com/microsoft/sample-app-aoai-chatGPT/issues/442

Once the new private endpoint was provisioned and approved, everything started to work.