Accessing stateless service via ServiceProxy fails + ASP.NET 5 Web API project throws Health State error

Question

I'm new to microsoft azure service fabric. For my master's degree I have to develop a microservice-approach prototype in service fabric. After hours of researching I am still not getting my issue(s) solved.

I want to access my (in a local fabric cluster deployed) stateless service in a web front-end like in https://azure.microsoft.com/en-us/documentation/articles/service-fabric-add-a-web-frontend/. The simplest way for doing that is by adding an ASP .NET 5 Web Api project to the Service Fabric application and make a ServiceProxy method call in the ValuesController. So I added this code to my solution:

ValuesController.cs:

[Route("api/[controller]")]
public class ValuesController : Controller
{
  // GET api/values/IObject
  [HttpGet("{interfaceName}")]
  public async Task<string> Get(string interfaceName)
  {
    var serviceName = "fabric:/DataServiceFabric/MasterDataMService";
    var masterDataService = ServiceProxy.Create<IMasterDataMService>(new Uri(serviceName));
    var result = await masterDataService.GetMasterDataByName(interfaceName);
    return result.Content;
  }
}

After a F5-deploy my browser doesn't automatically navigate to my web front-end. By looking into the Service Fabric Explorer my ASP .NET 5 application throws a Health State error:

Kind        Health State  Description
=============================================================================
Partitions  Error         Unhealthy partitions: 100% (1/1), MaxPercentUnhealthyPartitionsPerService=0%.
Partition   Error         Unhealthy partition: PartitionId='413...', AggregatedHealthState='Error'.
Event       Error         Error event: SourceId='System.FM', Property='State'. Partition is below target replica or instance count.

After this this question the "Partition is below target replica or instance count" indicates that a unhandled exception in my service is preventing it from starting. But I'm not able to find a stack strace in my Service Fabric Explorer to debug this failure. This is my ServiceManifest.xml of my ASP .NET web service:

ServiceManifest.xml (Web1):

<?xml version="1.0" encoding="utf-8"?>
<ServiceManifest xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" Name="Web1" Version="1.0.0" xmlns="http://schemas.microsoft.com/2011/01/fabric">
   <ServiceTypes>
      <StatelessServiceType ServiceTypeName="Web1Type">
         <Extensions>
            <Extension Name="__GeneratedServiceType__">
               <GeneratedNames xmlns="http://schemas.microsoft.com/2015/03/fabact-no-schema">
                  <DefaultService Name="Web1Service" />
                  <ServiceEndpoint Name="Web1TypeEndpoint" />
               </GeneratedNames>
            </Extension>
         </Extensions>
      </StatelessServiceType>
   </ServiceTypes>
   <CodePackage Name="C" Version="1.0.0">
      <EntryPoint>
         <ExeHost>
            <Program>approot\runtimes\dnx-clr-win-x64.1.0.0-rc1-update1\bin\dnx.exe</Program>
            <Arguments>--appbase approot\src\Web1 Microsoft.Dnx.ApplicationHost Microsoft.ServiceFabric.AspNet.Hosting --server Microsoft.AspNet.Server.WebListener</Arguments>
            <WorkingFolder>CodePackage</WorkingFolder>
            <ConsoleRedirection FileRetentionCount="5" FileMaxSizeInKb="2048" />
         </ExeHost>
      </EntryPoint>
   </CodePackage>
   <Resources>
      <Endpoints>
         <Endpoint Name="Web1TypeEndpoint" Protocol="http" Type="Input" Port="80" />
      </Endpoints>
   </Resources>
</ServiceManifest>

And here my ApplicationManifest.xml of my service fabric solution:

ApplicationManifest.xml:

<?xml version="1.0" encoding="utf-8"?>
<ApplicationManifest xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" ApplicationTypeName="DataServiceFabricType" ApplicationTypeVersion="1.0.0" xmlns="http://schemas.microsoft.com/2011/01/fabric">
   <Parameters>
      <Parameter Name="ActorTestServiceActorService_PartitionCount" DefaultValue="10" />
      <Parameter Name="MasterDataMService_InstanceCount" DefaultValue="-1" />
   </Parameters>
   <ServiceManifestImport>
     <ServiceManifestRef ServiceManifestName="Web2Pkg" ServiceManifestVersion="1.0.0" />
     <ConfigOverrides />
   </ServiceManifestImport>
   <ServiceManifestImport>
      <ServiceManifestRef ServiceManifestName="Web1" ServiceManifestVersion="1.0.0" />
   </ServiceManifestImport>
   <ServiceManifestImport>
      <ServiceManifestRef ServiceManifestName="ActorTestServicePkg" ServiceManifestVersion="1.0.0" />
   </ServiceManifestImport>
   <ServiceManifestImport>
      <ServiceManifestRef ServiceManifestName="MasterDataMServicePkg" ServiceManifestVersion="1.0.0" />
      <ConfigOverrides />
   </ServiceManifestImport>
   <DefaultServices>
      <Service Name="Web1Service">
         <StatelessService ServiceTypeName="Web1Type">
            <SingletonPartition />
         </StatelessService>
      </Service>
      <Service Name="ActorTestServiceActorService" GeneratedIdRef="761ee3cf-5a3a-49d8-9c57-aa3480d1acf1">
         <StatelessService ServiceTypeName="ActorTestServiceActorServiceType">
            <UniformInt64Partition PartitionCount="[ActorTestServiceActorService_PartitionCount]" LowKey="-9223372036854775808" HighKey="9223372036854775807" />
         </StatelessService>
      </Service>
      <Service Name="MasterDataMService">
         <StatelessService ServiceTypeName="MasterDataMServiceType" InstanceCount="[MasterDataMService_InstanceCount]">
            <SingletonPartition />
         </StatelessService>
      </Service>
   </DefaultServices>
</ApplicationManifest>

So I created a new solution with an ASP.NET 5 web application and the same ValuesController.cs. I ensured my stateless service is running on my local cluster and than I started my new web application. After calling the GET-Method in my Controller I got the following exception:

Exception thrown: 'System.Fabric.FabricException' in mscorlib.dll
Microsoft.AspNet.Hosting.Internal.HostingEngine: Information: Request finished in 0,2593ms 500
Microsoft.AspNet.Server.Kestrel: Error: An unhandled exception was thrown by the application.
System.Fabric.FabricException: Invalid partition key/ID '{0}'  for selector {1}

My stateless service is a SingletonPartition, so do I need a partition key here? And if yes, how do I get the key? The Service Fabric Explorer doesn't provide me with this information for my stateless service. Here is the ServiceManifest.xml of my stateless service:

ServiceManifest.xml (MasterDataMService):

<?xml version="1.0" encoding="utf-8"?>
<ServiceManifest Name="MasterDataMServicePkg"
                 Version="1.0.0"
                 xmlns="http://schemas.microsoft.com/2011/01/fabric"
                 xmlns:xsd="http://www.w3.org/2001/XMLSchema"
                 xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  <ServiceTypes>
    <!-- This is the name of your ServiceType. 
         This name must match the string used in RegisterServiceType call in Program.cs. -->
    <StatelessServiceType ServiceTypeName="MasterDataMServiceType" />
  </ServiceTypes>

  <!-- Code package is your service executable. -->
  <CodePackage Name="Code" Version="1.0.0">
    <EntryPoint>
      <ExeHost>
        <Program>MasterDataMService.exe</Program>
      </ExeHost>
    </EntryPoint>
  </CodePackage>

  <!-- Config package is the contents of the Config directoy under PackageRoot that contains an 
       independently-updateable and versioned set of custom configuration settings for your service. -->
  <ConfigPackage Name="Config" Version="1.0.0" />

  <Resources>
    <Endpoints>
      <!-- This endpoint is used by the communication listener to obtain the port on which to 
           listen. Please note that if your service is partitioned, this port is shared with 
           replicas of different partitions that are placed in your code. -->
      <Endpoint Name="ServiceEndpoint" Type="Input" Protocol="http" Port="80"/>
    </Endpoints>
  </Resources>
</ServiceManifest>

After that I decided to set up a service communication with OWIN:

MasterDataMService.cs:

internal sealed class MasterDataMService : StatelessService, IMasterDataMService
{
  [...]      

  protected override IEnumerable<ServiceInstanceListener> CreateServiceInstanceListeners()
  {
    return new[]
    {
      new ServiceInstanceListener(initParams => new OwinCommunicationListener("MasterDataMService", new StartUp(), initParams))
    };
  }
}

Now I can acess my microservice by using a HttpClient in my DefaultController:

var client = new HttpClient();
var request = "http://localhost:80/MasterDataMService/api/values/query";
var result = string.Empty;
HttpResponseMessage response = await client.GetAsync(request);
if (response.IsSuccessStatusCode)
{
  result = await response.Content.ReadAsStringAsync();
}

But thats not what I originally wanted. I don't want to specifiy the service endpoint in my request. Instead I would like to communicate with my stateless service over a ServiceProxy. How do I achieve that here? What did I wrong? And how can I solve this Health State error with my ASP .NET 5 application which is deployed into my service fabric cluster?

Thanks for your time.

Edit:

Extended stacktrace of invalid partition key exception:

Exception thrown: 'System.Fabric.FabricException' in mscorlib.dll
Microsoft.AspNet.Hosting.Internal.HostingEngine: Information: Request finished in 1,35ms 500
Microsoft.AspNet.Server.WebListener.MessagePump: Error: ProcessRequestAsync
System.Fabric.FabricException: Invalid partition key/ID '{0}'  for selector {1} ---> System.Runtime.InteropServices.COMException: exception of HRESULT: 0x80071BBF
   at System.Fabric.Interop.NativeClient.IFabricServiceManagementClient4.EndResolveServicePartition(IFabricAsyncOperationContext context)
   at System.Fabric.FabricClient.ServiceManagementClient.ResolveServicePartitionEndWrapper(IFabricAsyncOperationContext context)
   at System.Fabric.Interop.AsyncCallOutAdapter2`1.Finish(IFabricAsyncOperationContext context, Boolean expectedCompletedSynchronously)
   --- End of inner exception stack trace ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.ServiceFabric.Services.Client.ServicePartitionResolver.<ResolveAsyncHelper>d__2a.MoveNext()
--- End of stack trace from the previous location where the exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.ServiceFabric.Services.Communication.Client.CommunicationClientFactoryBase`1.<GetClientAsync>d__a.MoveNext()

Please give me feedback if you need more. (full stack trace is 82 lines long)

Invalid scheme exception stack trace:

Exception thrown: 'System.ArgumentException' in mscorlib.dll
Microsoft.AspNet.Hosting.Internal.HostingEngine: Information: Request finished in 1,45ms 500
Microsoft.AspNet.Server.WebListener.MessagePump: Error: ProcessRequestAsync
System.ArgumentException: the provided uri scheme 'http' is invalid; expected 'net.tcp'.
Parametername: via
   at System.ServiceModel.Channels.TransportChannelFactory`1.ValidateScheme(Uri via)
   at System.ServiceModel.Channels.ConnectionOrientedTransportChannelFactory`1.OnCreateChannel(EndpointAddress address, Uri via)
   at System.ServiceModel.Channels.ChannelFactoryBase`1.InternalCreateChannel(EndpointAddress address, Uri via)
   at System.ServiceModel.Channels.ServiceChannelFactory.ServiceChannelFactoryOverDuplexSession.CreateInnerChannelBinder(EndpointAddress to, Uri via)
   at System.ServiceModel.Channels.ServiceChannelFactory.CreateServiceChannel(EndpointAddress address, Uri via)
   at System.ServiceModel.Channels.ServiceChannelFactory.CreateChannel(Type channelType, EndpointAddress address, Uri via)
   at System.ServiceModel.DuplexChannelFactory`1.CreateChannel(InstanceContext callbackInstance, EndpointAddress address, Uri via)
   at System.ServiceModel.DuplexChannelFactory`1.CreateChannel(InstanceContext callbackInstance, Binding binding, EndpointAddress endpointAddress)
   at Microsoft.ServiceFabric.Services.Communication.Wcf.Client.WcfCommunicationClientFactory`1.<CreateClientAsync>d__2.MoveNext()
--- End of stack trace from the previous location where the exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.ServiceFabric.Services.Communication.Client.CommunicationClientFactoryBase`1.<CreateClientWithRetriesAsync>d__1e.MoveNext()

Answer 1

I ran into the same problem while learning about Service Fabric. Turned out that providing just the URI was not enough - I also had to specify the partition key as a magic value of one:

IHelloService service = ServiceProxy.Create<IHelloService>(new Uri("fabric:/Application1/HelloService"), new ServicePartitionKey(1));

Kudos to this thread on disq.us. There is also a deeper explanation provided by a Microsoft engineer Oana Platon as to why the value of 1 works:

Diogo, take a look at this article that explains partitioning: link Specifically, look at ranged partitioning (otherwise known as UniformInt64Partition): "This is used to specify an integer range (identified by a low key and high key) and a number of partitions (n). It creates n partitions, each responsible for a non-overlapping subrange of the overall partition key range. For example, a ranged partitioning scheme with a low key of 0, a high key of 99, and a count of 4 would create four partitions, as shown below." Then look at your service manifest and figure out how it is configured - how many partitions and what is the range (low key - high key). If you have one partition, any key in that range goes to the (one) partition, so it doesn't matter which key you specify. If you have more than one partition, you need to figure out with which one your client needs to talk to. Specify a partition key in the range that partition is serving.

I must admit that I myself must study partitioning in greater depth to understand this explanation.

Answer 2

In my case, none of the above worked. Or more precisely, my solution was a combination/variation of the above solutions

I needed to do 2 things: First, make sure I have registered my services:

  protected override IEnumerable<ServiceReplicaListener> CreateServiceReplicaListeners()
    {
        return new[] { new ServiceReplicaListener(context => this.CreateServiceRemotingListener(context)) };}

And second, refer to the very first service ("magic" value of 0)

        ServiceProxy.Create<IMasterDataMService>(new Uri("fabric:/DataServiceFabric/MasterDataMService"), new ServicePartitionKey(0));

I hope this helps

Answer 3

I ran in to exactly the same problem using ServiceProxy with a Stateless Service. I have a Stateful Services that works well, but the Stateless was receiving:

System.Fabric.FabricException: Invalid partition key/ID '{0}'  for selector {1}

As Allan T mentions above, you also need to use the third constructor overload for ServiceProxy.Create():

Instead of:

ServiceProxy.Create<IMasterDataMService>(0, new Uri("fabric:/DataServiceFabric/MasterDataMService"));

use:

ServiceProxy.Create<IMasterDataMService>(new Uri("fabric:/DataServiceFabric/MasterDataMService"));

...simply, do not specify the partition. That made the error go away for me.

https://msdn.microsoft.com/en-us/library/mt628402.aspx

Answer 4

In the getting started examples, there is a class ServiceUriBuilder. Initialize this class by passing the service name in the constructor.

var proxyLocation = new ServiceUriBuilder("MasterDataMService");
var masterDataService = ServiceProxy.Create<IMasterDataMService>(proxyLocation.ToUri());

var result = await masterDataService.GetMasterDataByName(interfaceName);

Also in your MasterDataMService CreateServiceInstanceListeners method, make sure it looks something like this:

    protected override IEnumerable<ServiceInstanceListener> CreateServiceInstanceListeners()
    {
        return new[] { new ServiceInstanceListener(context => new FabricTransportServiceRemotingListener(context, this)) };
    }

Answer 5

Judging by your second stack trace:

System.ArgumentException: the provided uri scheme 'http' is invalid; expected 'net.tcp'.

It looks like you're trying to use ServiceProxy to connect to a service that is listening on an HTTP endpoint. ServiceProxy expects a Service Remoting Listener which uses a binary protocol, not HTTP.

The first exception is still a bit of a mystery:

System.Fabric.FabricException: Invalid partition key/ID '{0}'  for selector {1}

This will only happen if you attempt to resolve a uniform int64 or named partitioned service without providing a partition key. In your config it appears you have the target service set up as a singleton partition, in which case you should not see this error, so can you double-check that the service you are trying to connect to is in fact created as a singleton partition service?

Answer 6

I ran into these same frustrating issues myself and resolved them.

A couple of things to check

1) Make sure your ASP.NET 5 web api project is not referencing any X64 libraries.

2) Port Clash.. when deploying your ASPNET 5 web api to the Cluster, make sure there are no 2 websites that are using the same port. You can change the port inside the [ASP.NET 5 project]/PackageRoot/ServiceManifest.xml.. towards the bottom

**3) "wrap" folder madness! Verify that your wrap folder (at the root solution folder) only contains your .net 4.5.1 class libraries. My issues went away when I deleted the "Newtonsoft.Json" folder inside the wrap folder. You will then have to run dnu restore on the solution so it will recreate the project.lock.json files

4) Make sure nothing is blowing up inside your startup.cs class. Run the web api locally just to make sure its loading; its normal that the service proxy class will fail to load.