Service fabric taking time to deploy RollingForwardPending taking time

Question

Service Fabric is taking time after each stage of deploying to upgradedomain it it spends 3min in RollingForwardPending before going to next upgradedomain. Is there a way to reduce this?

2018-12-25T02:30:38.5014666Z Domain Wise Upgrade Status:  UpgradeDomainName=0; State=InProgress;  UpgradeDomainName=1; State=Pending;  UpgradeDomainName=2; State=Pending;
2018-12-25T02:30:43.5070541Z Current Upgrade State: RollingForwardInProgress
2018-12-25T02:30:48.5186098Z Current Upgrade State: RollingForwardInProgress
2018-12-25T02:30:53.5258935Z Current Upgrade State: RollingForwardInProgress
2018-12-25T02:30:58.5340237Z Current Upgrade State: RollingForwardInProgress
2018-12-25T02:31:03.5350754Z Current Upgrade State: RollingForwardInProgress
2018-12-25T02:31:08.5462591Z Current Upgrade State: RollingForwardInProgress
2018-12-25T02:31:13.5592646Z Current Upgrade State: RollingForwardInProgress
2018-12-25T02:31:18.5656780Z Current Upgrade State: RollingForwardInProgress
2018-12-25T02:31:23.7048413Z Current Upgrade State: RollingForwardInProgress
2018-12-25T02:31:28.7162008Z Current Upgrade State: RollingForwardInProgress
2018-12-25T02:31:33.7272212Z Current Upgrade State: RollingForwardInProgress
2018-12-25T02:31:38.7354639Z Current Upgrade State: RollingForwardInProgress
2018-12-25T02:31:43.7385037Z Current Upgrade State: RollingForwardInProgress
2018-12-25T02:31:48.7558681Z Current Upgrade State: RollingForwardInProgress
2018-12-25T02:31:53.7610563Z Current Upgrade State: RollingForwardInProgress
2018-12-25T02:31:58.7825059Z Current Upgrade State: RollingForwardInProgress
2018-12-25T02:32:03.7911006Z Current Upgrade State: RollingForwardInProgress
2018-12-25T02:32:08.8379252Z Current Upgrade State: RollingForwardInProgress
2018-12-25T02:32:13.8531978Z Current Upgrade State: RollingForwardInProgress
2018-12-25T02:32:18.8619929Z Current Upgrade State: RollingForwardInProgress
2018-12-25T02:32:23.8601467Z Current Upgrade State: RollingForwardInProgress
2018-12-25T02:32:28.8755147Z Current Upgrade State: RollingForwardInProgress
2018-12-25T02:32:33.8749331Z Current Upgrade State: RollingForwardInProgress
2018-12-25T02:32:38.8863708Z Current Upgrade State: RollingForwardPending
2018-12-25T02:32:38.8876575Z Domain Wise Upgrade Status:  UpgradeDomainName=0; State=Completed;  UpgradeDomainName=1; State=Pending;  UpgradeDomainName=2; State=Pending;
2018-12-25T02:32:43.8911107Z Current Upgrade State: RollingForwardPending
2018-12-25T02:32:48.8990487Z Current Upgrade State: RollingForwardPending
2018-12-25T02:32:53.8986271Z Current Upgrade State: RollingForwardPending
2018-12-25T02:32:58.9033581Z Current Upgrade State: RollingForwardPending
2018-12-25T02:33:03.9116147Z Current Upgrade State: RollingForwardPending
2018-12-25T02:33:08.9232562Z Current Upgrade State: RollingForwardPending
2018-12-25T02:33:13.9309302Z Current Upgrade State: RollingForwardPending
2018-12-25T02:33:18.9412184Z Current Upgrade State: RollingForwardPending
2018-12-25T02:33:23.9446965Z Current Upgrade State: RollingForwardPending
2018-12-25T02:33:28.9565137Z Current Upgrade State: RollingForwardPending
2018-12-25T02:33:33.9679990Z Current Upgrade State: RollingForwardPending
2018-12-25T02:33:38.9793697Z Current Upgrade State: RollingForwardPending
2018-12-25T02:33:43.9839637Z Current Upgrade State: RollingForwardPending
2018-12-25T02:33:48.9898980Z Current Upgrade State: RollingForwardPending
2018-12-25T02:33:53.9988117Z Current Upgrade State: RollingForwardPending
2018-12-25T02:33:59.0034323Z Current Upgrade State: RollingForwardPending
2018-12-25T02:34:04.0111436Z Current Upgrade State: RollingForwardPending
2018-12-25T02:34:09.0177974Z Current Upgrade State: RollingForwardPending
2018-12-25T02:34:14.0189917Z Current Upgrade State: RollingForwardPending
2018-12-25T02:34:19.0206275Z Current Upgrade State: RollingForwardPending
2018-12-25T02:34:24.0221773Z Current Upgrade State: RollingForwardPending
2018-12-25T02:34:29.0297897Z Current Upgrade State: RollingForwardPending
2018-12-25T02:34:34.0417619Z Current Upgrade State: RollingForwardPending
2018-12-25T02:34:39.0528795Z Current Upgrade State: RollingForwardInProgress
2018-12-25T02:34:39.0541447Z Domain Wise Upgrade Status:  UpgradeDomainName=1; State=InProgress;  UpgradeDomainName=0; State=Completed;  UpgradeDomainName=2; State=Pending;

Answer 1

For me the deployment task completes faster if that application is not currently deployed to the Service Fabric cluster.

So, the overall process should be much faster if you:

1. Delete the old application from your cluster (1st)
2. Deploy the new package (2nd)

---

#  Method          Task Time
----------------------------
1. Upgrade        10 minutes
2. Delete First   50 seconds

Answer 2

According to this doc for Service Fabric Troubleshooting guides, Service Fabric handles cluster wide settings changes such as Security changes, Placement Settings, custom fabric settings, etc as a cluster upgrade and as such it will trigger a two phase full UD (Upgrade Domain) walk to apply these changes to the cluster one upgrade domain at a time. After the changes are applied it will wait for some period of time based on the configured health and stability settings to ensure the change does not cause your cluster to destabilize.

Also, please refer to this doc for Troubleshoot application upgrades

There is a MSDN post for this issue. Take a look here.

Answer 3

Use the appropriate upgrade mode: https://learn.microsoft.com/en-us/azure/service-fabric/service-fabric-application-upgrade#upgrade-modes

For you scenario and unmonitored auto might be appropriate: https://learn.microsoft.com/en-us/azure/service-fabric/service-fabric-application-upgrade-parameters

Indicates that the upgrade mode is unmonitored automatic. After Service Fabric upgrades an upgrade domain, Service Fabric upgrades the next upgrade domain irrespective of the application health state. This mode is not recommended for production, and is only useful during development of an application.

Also check these parameters: HealthCheckStableDurationSec and HealthCheckWaitDurationSec