Cannot register on-prem Self Hosted Integration Runtime (SHIR) with Azure Data Factory (ADF) through a Private Endpoint (PE) - InvalidGatewayKey error

Question

My datafactory has an AutoResolver Azure IR running in a managed private network, and I have various managed private endpoints out to various other Azure services - KeyVault, Storage Account, Function Apps (with associated linked services configured). This is all working fine.

The datafactory has public network access disabled.

On my virtual network subnet, I have 2x Private Endpoints configured for my datafactory. One PE is of sub-type factory, the other PE is of sub-type portal.

(I don't think this is relevant - but just FYI I have other Private Endpoints pointing to the same other PaaS resources I've linked to via managed private endpoints within the ADF)

I've installed a SHIR on an on-prem server. From this server's CLI I'm able to resolve the ADF's FQDN back to the correct Private Endpoint IP address in my Azure virtual network subnet.

When attempting to register the SHIR, it fails with this InvalidGatewayKey error:

Failed to get service token from ADF service with key IR@61[redacted]f6@[redacted]-df-shared@ServiceEndpoint=[redacted]-df-shared.[region-redacted].datafactory.azure.net@****************** and time cost is: 0.1606498 seconds, the error code is: InvalidGatewayKey, activityId is: e14a4ef0-f581-4396-bae8-f2a3d680435b and detailed error message is Processed HTTP request failed. .

Running the troubleshooter and looking at the logs, I can see it does resolve the FQDN back to the private endpoint IP address correctly, and it gets the server certificate, but then it fails to authenticate.

There are posts all over the web and SO about this type of error - but it usually relates to DNS issues (and can be fixed with a hosts file entry on the SHIR OS). An example on SO I found: Cannot register SHIR(ADF) on Azure VM

If I enable public access, I'm able to register the SHIR fine. But then when I disable public access, the SHIR can no longer communicate.

My network infrastructure team advises that they're seeing no outgoing traffic being blocked at the firewall from this server to the Private Endpoint address.

Any and all help appreciated! Let me know if you need further info.

Full log (minus redacted info) from the troubleshooter is

DNS resolve
0.006s
Result: Success
Documentation: https://go.microsoft.com/fwlink/?linkid=2166762
[UTC 2/26/2025 4:12:21 AM] DNS resolve: [redacted]-df-shared.[region-redacted].datafactory.azure.net
[UTC 2/26/2025 4:12:21 AM] GetHostEntry([redacted]-df-shared.[region-redacted].datafactory.azure.net) returns:
[UTC 2/26/2025 4:12:21 AM]     [first-two-octets-redacted].12.8
 
Get service endpoint certificate
0.110s
Result: Success
Documentation: https://go.microsoft.com/fwlink/?linkid=2166762
[UTC 2/26/2025 4:12:21 AM] Get service endpoint certificate information: https://[redacted]-df-shared.[region-redacted].datafactory.azure.net/
[UTC 2/26/2025 4:12:21 AM] Get certificate Hash: C9[redacted]D3
 
Connect to cloud service endpoint
0.169s
Result: Failed
Error Message: Failed to get service token from ADF service with key IR@61[redacted]f6@[redacted]-df-shared@ServiceEndpoint=[redacted]-df-shared.[region-redacted].datafactory.azure.net@****************** and time cost is: 0.1606498 seconds, the error code is: InvalidGatewayKey, activityId is: e14a4ef0-f581-4396-bae8-f2a3d680435b and detailed error message is Processed HTTP request failed. .
Parameters:
Endpoint: https://[redacted]-df-shared.[region-redacted].datafactory.azure.net/
Documentation: https://go.microsoft.com/fwlink/?linkid=2166762
[UTC 2/26/2025 4:12:21 AM] Connecting to cloud service endpoint: https://[redacted]-df-shared.[region-redacted].datafactory.azure.net/
{
  "ClassName": "Microsoft.DataTransfer.DIAgentClient.HostServiceException",
  "Message": "Failed to get service token from ADF service with key IR@61[redacted]f6@[redacted]-df-shared@ServiceEndpoint=[redacted]-df-shared.[region-redacted].datafactory.azure.net@****************** and time cost is: 0.1606498 seconds, the error code is: InvalidGatewayKey, activityId is: e14a4ef0-f581-4396-bae8-f2a3d680435b and detailed error message is Processed HTTP request failed.\r\n.",
  "Data": {
    "ErrorCode": 15
  },
  "InnerException": null,
  "HelpURL": null,
  "StackTraceString": "   at Microsoft.DataTransfer.DiagnosticTool.ServiceFrontendDomainConnectionSuite.d__8.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.DataTransfer.DiagnosticTool.DiagnosticRunner.d__8.MoveNext()",
  "RemoteStackTraceString": null,
  "RemoteStackIndex": 0,
  "ExceptionMethod": "8\nMoveNext\nMicrosoft.DataTransfer.DiagnosticTool, Version=5.48.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35\nMicrosoft.DataTransfer.DiagnosticTool.ServiceFrontendDomainConnectionSuite+d__8\nVoid MoveNext()",
  "HResult": -2146233088,
  "Source": "Microsoft.DataTransfer.DiagnosticTool",
  "WatsonBuckets": null
}