Service Fabric - Stateful Service Persistence

Question

I am new to service fabric and started by looking at the MSDN articles covering the topic. I began by implementing the Hello World sample here.

I changed their original RunAsync implementation to:

var myDictionary = await this.StateManager.GetOrAddAsync<IReliableDictionary<int, DataObject>>("myDictionary");

while (!cancellationToken.IsCancellationRequested)
{
    DataObject dataObject;

    using (var tx = this.StateManager.CreateTransaction())
    {
        var result = await myDictionary.TryGetValueAsync(tx, 1);

        if (result.HasValue)
            dataObject = result.Value;
        else
            dataObject = new DataObject();
        //
        dataObject.UpdateDate = DateTime.Now;
        //
        //ServiceEventSource.Current.ServiceMessage(
        //    this,
        //    "Current Counter Value: {0}",
        //    result.HasValue ? result.Value.ToString() : "Value does not exist.");

        await myDictionary.AddOrUpdateAsync(tx, 1, dataObject, ((k, o) => dataObject));

        await tx.CommitAsync();
    }
    await Task.Delay(TimeSpan.FromSeconds(1), cancellationToken);
}

I also introduced a DataObject type and have exposed an UpdateDate property on that type.

[DataContract(Namespace = "http://www.contoso.com")]
public class DataObject
{
    [DataMember]
    public DateTime UpdateDate { get; set; }
}

When I run the app (F5 in visual studio 2015), a dataObject instance (keyed as 1) is not found in the dictionary so I create one, set UpdateDate, add it to the dictionary and commit the transaction. During the next loop, it finds the dataObject (keyed as 1) and sets UpdateDate, updates the object in the dictionary and commits the transaction. Perfect.

Here's my question. When I stop and restart the service project (F5 in visual studio 2015) I would expect that on my first iteration of the RunAsync that the dataObject (keyed as 1) would be found but it's not. I would expect all state to be flushed to its replica.

Do I have to do anything for the stateful service to flush its internal state to its primary replica?

From what I've read, it makes it sound as though all of this is handled by service fabric and that calling commit (on the transaction) is sufficient. If I locate the primary replica (in Service Fabric Explorer->Application View) I can see that the RemoteReplicator_xxx LastACKProcessedTimeUTC is updated once I commit the transaction (when stepping through).

Any help is greatly appreciated.

Thank you!

-Mark

Answer 1

You need to think of a ReliableDictionary as holding collections of objects as opposed to a collection of references. That is, when you add an “object” to the dictionary, you must think that you are handing the object off completely; and you must not alter this object’s state in the anymore. When you ask ReliableDictionary for an “object”, it gives you back a reference to its internal object. The reference is returned for performance reasons and you are free to READ the object’s state. (It would be great if the CLR supported read-only objects but it doesn't.) However, you MUST NOT MODIFY the object’s state (or call any methods that would modify the object’s state) as you would be modifying the internal data structures of the dictionary corrupting its state.

To modify the object’s state, you MUST make a copy of the object pointed to by the returned reference. You can do this by serializing/deserializing the object or by some other means (such as creating a whole new object and copying the old state to the new object). Then, you write the NEW OBJECT into the dictionary. In a future version of Service Fabric, We intend to improve ReliableDictionary’s APIs to make this required pattern of use more discoverable.

Answer 2

This is a function of the default local development experience in Visual Studio. If you watch the Output window closely after hitting F5 you'll see a message like this:

The deployment script detects that there's an existing app of the same type and version already registered, so it removes it and deploys the new one. In doing that, the data associated with the old application is removed.

You have a couple of options to deal with this.

In production, you would perform an application upgrade to safely roll out the updated code while maintaining the state. But constantly updating your versions while doing quick iteration on your dev box can be tedious.

An alternative is to flip the project property "Preserve Data on Start" to "Yes". This will automatically bump all versions of the generated application package (without touching the versions in your source) and then perform an app upgrade on your behalf.

Note that because of some of the system checks inherent in the upgrade path, this deployment option is likely to be a bit slower than the default remove-and-replace. However, when you factor in the time it takes to recreate the test data, it's often a wash.