Intermittent "Validation of ViewState MAC Failed" Error in SSRS Scale-Out Deployment Despite Correct Machine Key Configuration

Question

I am experiencing an intermittent "Validation of ViewState MAC failed" error in my SQL Server Reporting Services (SSRS) scale-out deployment. The error message indicates:

Error rending control: System.Web.HttpException (0x80004005): Validation of viewstate MAC failed. If this application is hosted by a Web Farm or cluster, ensure that <machineKey> configuration specifies the same validationKey and validation algorithm. AutoGenerate cannot be used in a cluster.

Despite thoroughly verifying and configuring the settings, the issue persists. The error occurs intermittently and can happen on either of the two Azure VMs hosting SSRS. The VMs generally work fine, but occasionally, the error appears on one node or the other without any identifiable pattern.

Here’s a summary of my current setup and troubleshooting steps taken so far:

1. Machine Key Configuration:

   • Both SSRS nodes have identical <machineKey> settings in their RSReportServer.config file.
   • The ValidationKey and DecryptionKey values are explicitly defined and consistent across nodes.
   • The validation algorithm is set to a secure and matching configuration (SHA1).

2. Load Balancer Configuration:

   • We use Azure Load Balancer, which operates at Layer 4 (TCP) and has session persistence options of None, Client IP, and Client IP and Protocol.
   • Session Persistence: Currently, the session persistence is set to None, as the load balancer operates on the transport layer and does not support affinity based on HTTP session state.
   • I cannot configure session persistence as Client IP because all requests originate from the same source: our web application. As a result, using Client IP-based affinity would not effectively distribute traffic.

3. SSRS Service Architecture:

   • SSRS is configured as a scale-out deployment and does not rely on IIS.
   • Both SSRS nodes run as standalone services, and application domain recycling settings are at their default values.
   • The error occurs intermittently on either node, and there is no discernible pattern to when or why the error happens.

4. Additional Settings and Observations:

   • System clocks on both nodes are synchronized.
   • There are no apparent network latency issues or load balancer timeouts.
   • The application domain recycling interval has not been adjusted from the default setting of 12 hours.

Questions:

1. What could be causing the ViewState MAC validation error despite the correct machine key configuration?
2. Are there any overlooked settings or SSRS-specific configurations that might contribute to this issue?
3. How can I troubleshoot this further to ensure the reliability of our SSRS deployment?

Any guidance or suggestions would be greatly appreciated!