JSON.Net deserialization of DateTime values failing in some instances?

Question

Using JSON.Net, of the following 5 tests, the first and last pass while the others fail:

[Test, Sequential]
public void WhyCantIDeserializeThisDateWhen2011Works(
    [Values(1980, 1980, 1980, 1980, 1980, 1980, 1980)] Int32 year,
    [Values(10,   10,   10,   10,   11,   11,   11)] Int32 month,
    [Values(26,   27,   30,   31,   1,    2,    3)] Int32 day)
{
    var obj = new {
        Title = "Will this be able to serialize the DateTime field?",
        Timestamp = new DateTime(year, month, day)
    };

    var type = obj.GetType();
    var serialized = Newtonsoft.Json.JsonConvert.SerializeObject(obj);
    dynamic deserialized = Newtonsoft.Json.JsonConvert.DeserializeObject(serialized, type);

    Assert.AreEqual(obj.Title, deserialized.Title);
    Assert.AreEqual(obj.Timestamp, deserialized.Timestamp);
}

Here is some of the output:

[snip]

Test 'Rds.Infrastructure.Serializers.Tests.JsonSerializerTests.WhyCantIDeserializeThisDateWhen2011Works(1980,11,2)' failed:
Expected: 1980-11-02 00:00:00.000
But was: 1980-11-01 23:00:00.000
at CallSite.Target(Closure , CallSite , Type , DateTime , Object )
    UnitTests\Rds.Infrastructure\Serializers\JsonSerializerTests.cs(141,0): at     Rds.Infrastructure.Serializers.Tests.JsonSerializerTests.WhyCantIDeserializeThisDateWhen2011Works(Int32 year, Int32 month, Int32 day)

2 passed, 5 failed, 0 skipped, took 627.55 seconds (NUnit 2.5.5).

The error is typical of all errors - when it reloads the date, rather than being the specified day, it comes up as the previous day at 11pm. This is particularly odd, in that if I change the year to 2011, all these tests pass.

I've dug into the JSON.Net code - the JsonTextReader class's ParseDate method reads in the value. Using Oct 27 2011 as an example, comments are mine:

private void ParseDate(string text)
{
  string value = text.Substring(6, text.Length - 8);
  DateTimeKind kind = DateTimeKind.Utc;

  int index = value.IndexOf('+', 1);

  if (index == -1)
    index = value.IndexOf('-', 1);

  TimeSpan offset = TimeSpan.Zero;

  if (index != -1)
  {
    kind = DateTimeKind.Local;
    offset = ReadOffset(value.Substring(index));
    value = value.Substring(0, index);
  }

  long javaScriptTicks = long.Parse(value, NumberStyles.Integer, CultureInfo.InvariantCulture);

  // The date time gets loaded here, as Oct 27 2011 3am
  DateTime utcDateTime = JsonConvert.ConvertJavaScriptTicksToDateTime(javaScriptTicks);

#if !NET20
  if (_readType == ReadType.ReadAsDateTimeOffset)
  {
    SetToken(JsonToken.Date, new DateTimeOffset(utcDateTime.Add(offset).Ticks, offset));
  }
  else
#endif
  {
    DateTime dateTime;

    switch (kind)
    {
      case DateTimeKind.Unspecified:
        dateTime = DateTime.SpecifyKind(utcDateTime.ToLocalTime(), DateTimeKind.Unspecified);
        break;
      case DateTimeKind.Local:
        // Here, it gets converted to local time, Oct 26 2011 at 11pm!
        dateTime = utcDateTime.ToLocalTime();
        break;
      default:
        dateTime = utcDateTime;
        break;
    }

    SetToken(JsonToken.Date, dateTime);
  }
}

As noted above, the error only occurs for dates from Oct 27 1980 through Nov 2 1980. I haven't run tests to determine which years are problems, but the tests do pass if you use 2011.

I'm guessing this has to do with changes in daylight savings time?

Does anyone have an idea of what's going on here?

Answer 1

I've dug into it much further.

When JSON.Net serializes a DateTime instance, it calls TimeZone.CurrentTimeZone.GetUtcOffset(dt). When it reloads the DateTime, it assumes a UTC date and to convert to a local date calls utcDateTime.ToLocalTime() to convert to local time. It appears these two methods may not always use the same offset:

(Note: I'm in the Atlantic Time Zone.)

[Test, Sequential]
public void AnotherTest(
    [Values(2006, 2006, 2006, 2006, 2006, 2006, 2006)] Int32 year,
    [Values(10, 10, 10, 10, 11, 11, 11)] Int32 month,
    [Values(26, 27, 30, 31, 1, 2, 3)] Int32 day)
{
    var dt = new DateTime(year, month, day, 0, 0, 0, DateTimeKind.Local);

    var utcOffset1 = TimeZone.CurrentTimeZone.GetUtcOffset(dt);
    var utcOffset2 = dt.Subtract(dt.ToUniversalTime());

    Assert.AreEqual(utcOffset1, utcOffset2);
}

These pass on Oct 26 and Nov 3, and fail for the dates between. I also ran tests for various earlier years, with same results. For dates from 2007 through 2011, these all pass. For all failures I've found, utcOffset1 was -3 hours while utcOffset2 was -4 hours. (According to http://www.timeanddate.com/library/abbreviations/timezones/, Atlantic Standard Time is supposed to be UTC-4 and Atlantic Daylight Time should be UTC-3.) A few quick tests has shown me that there is also a problem at the start of daylight savings time as well, prior to 2007.

(I've opened an issue at https://connect.microsoft.com/VisualStudio/feedback/details/699491/timezone-getutcoffset-and-datetime-touniversaltime-not-consistent-for-brief-period-for-atlantic-time-zone.)

In the meantime, to fix this issue, the UTC offset just needs to be made consistent when serializing and deserializing the date, which means getting rid of the call to TimeZone.CurrentTimeZone.GetUtcOffset.

UPDATE

JamesNK updated JSON.Net on November 1 2011 so that time zone conversions use TimeZoneInfo instead of TimeZone, which appears to have fixed the problem.

UPDATE 2

Thanks @derekhat for the following additional information:

Got a few free minutes tonight. All tests passed for me using Windows 7 64-bit and compiled at command line with .NET 2.0 SDK (had to change var to explicit type declarations).

5 of the 7 tests failed with Visual Studio 2010 and .NET 4.

I then found the following documentation.

The GetUtcOffset method recognizes only the current daylight saving time adjustment rule for the local time zone. As a result, it is guaranteed to accurately return the UTC offset of a local time only during the period in which the latest adjustment rule is in effect. It may return inaccurate results if Time is a historic date and time value that was subject to a previous adjustment rule.

The matter is further complicated by another piece of documentation that states: "On Windows XP systems, the ToUniversalTime method recognizes only the current adjustment rule when converting from local time to UTC." And a comment says the WinXP behaviour also exists on Windows Server 2003. The implication is that ToUniversalTime works properly with historic dates on newer versions of Windows, which seems to match your results.