Azure Service Fabric - deployment fails with 1 partitions remaining

Question

I've been using Service Fabric for a little while, successful built, deployed and tested several services, but the service which I've just finished building fails on deployment (see error below). In diagnosing I've tried creating and deploying a completely new service using the VS template (with no code changes) and that won't deploy either - the same error. However, the calculator SDK sample deploys just fine.

I get the error below where it fails to build one of the partitions:

Service Status:
fabric:/DataFabricServiceApplication/DataFabricService is not ready, 1 partitions remaining.

Service Status:
fabric:/DataFabricServiceApplication/DataFabricService is not ready, 1 partitions remaining.

Service Status:
fabric:/DataFabricServiceApplication/DataFabricService is not ready, 1 partitions remaining.

Service Status:
fabric:/DataFabricServiceApplication/DataFabricService is not ready, 1 partitions remaining.

Service Status:,fabric:/DataFabricServiceApplication/DataFabricService is not ready, 1 partitions remaining.

Something is taking too long, the application is still not ready.
Finished executing script 'Get-FabricApplicationStatus.ps1'.
Time elapsed: 00:01:48.0681346
The thread 0x37fc has exited with code 0 (0x0).
The thread 0x4fe4 has exited with code 0 (0x0).

Getting the health of the service gives the following:

    PS C:\WINDOWS\system32> Get-ServiceFabricServiceHealth -ServiceName fabric:/DataFabricServiceApplication/DataFabricService


    ServiceName           : fabric:/DataFabricServiceApplication/DataFabricService
    AggregatedHealthState : Error
    UnhealthyEvaluations  :
                            Unhealthy partitions: 100% (1/1), MaxPercentUnhealthyPartitionsPerService=0%.

                            Unhealthy partition: PartitionId='3eebd943-097d-4568-ad7e-d37c621a888b', AggregatedHealthState='Error'.

                                Error event: SourceId='System.FM', Property='State'.

    PartitionHealthStates :
                            PartitionId           : 3eebd943-097d-4568-ad7e-d37c621a888b
                            AggregatedHealthState : Error

    HealthEvents          :
                            SourceId              : System.FM
                            Property              : State
                            HealthState           : Ok
                            SequenceNumber        : 10
                            SentAt                : 11/11/2015 07:16:02
                            ReceivedAt            : 11/11/2015 07:16:03
                            TTL                   : Infinite
                            Description           : Service has been created.
                            RemoveWhenExpired     : False
                            IsExpired             : False
                            Transitions           : Warning->Ok = 11/11/2015 07:16:03, LastError = 01/01/0001 00:00:00

Does anyone know why this won't deploy / how I can get to the bottom of this? I've seen others with the same error but didn't find any resolution.

Answer 1

In my case the url for the service was wrong,I have typed fabric:/XXX.User.Application/XXX.Services.UserService

as

fabric:/XXX.User.Application/XXX.Services.User

Answer 2

Expanding on what @MarkD had said I ran into the same issue and a quick solution for me was to just push the code to my repository and then delete/re-clone it. Upon downloading the clone and rebuilding it I had no issues with deployment.

Answer 3

Catch/Log the errors in CreateServiceReplicaListeners method..

1. may be you updated the publish profiles for local (dev) service fabric cluster ensure that you are using the correct profile for local (dev) cluster ApplicationParameters in Local.1Node and Local.5Node

2. The endpoint not exist in the ServiceManifest

3. The Listenersettings are wrong.. etc.

Answer 4

Not an answer but I may be able to save you some time. I have this error on one machine (my laptop) and another running exactly the same code in an Azure VM is completely fine.

Short answer - I now run all my debug on a VM from a template I constructed which has WS2012R2/VS2015 and SF SDK. Every time I hit this bug, I commit the sourcecode to the team server, delete the VM, re-create it, get the source back and keep going. Takes about 10 minutes and has saved a lot of time on these pesky problems (no complaints - this is not unusual at pre-release/preview stage).

I suspect during debug/rebuild/debug/rebuild cycle that something corrupts in SF. I have tried resetting, remove/reinstall all the way to removing the SDKs and VS but nothing clears the error once it starts. Maybe there is a registry corruption of some sort that does not get reset on reinstall? I remember seeing one of the traces complaining about performance counter missing and another user spotted the same thing but I was unable to trace it any further. I'll try to look a little more once it happens again.

Answer 5

The unhealthy evaluations on service health show you the partition that has a problem. You can continue digging into the health of that partition (Get-ServiceFabricPartitionHealth 3eebd943-097d-4568-ad7e-d37c621a888b). My guess is that you will see an Error event from System.FM saying that the partition is below the min replica set size.

Then you can dig into the replicas health (Get-ServiceFabricReplica 3eebd943-097d-4568-ad7e-d37c621a888b | Get-ServiceFabricReplicaHealth). I've seen many cases when this happens because replica can't open (either because of a configuration issue or a bug in the service replica code). If that's the case, you will either see an event on the replicas saying what is wrong with it (eg. open takes a long time or open failed with an error code etc) or the replicas will keep recycling.