Asp.NET Core OpenIddict invalid_grant

Question

First thing first: I know this is large post, but I am tracking this problem for several weeks, and I collect a lot of information what may be the source of problem.

I am using Angular2 application with OpenIddict authentication. I get access_token, refresh_token on client app. I can use refresh_token to get new access_token, everything works. Almost.

At some point I get error response from server:

POST https://mydomain:2000/api/authorization/token 400 (Bad Request)

and response:

error:"invalid_grant"
error_description:"Invalid ticket"

I triple checked everything and the refresh_token I send is correct.

About design:
Before I make request to server I check if access_token expire. If expire, I send request to get new access_token with refresh_token.

And it works for random time, but on some random time(repeat) refresh_token becomes invalid.
I though it has something to do with AddEphemeralSigningKey, and I change it to AddSigningCertificate. (Details are in this thread.)

I think, that IIS kills Kestrel after some time of in-activity. My Application Pool config is:

StartMode: OnDemand
Idle Time-out (minutes): 20
Idle Time-out (action): Terminate

I suspect that after new request is made, OpenIddict wrongly de-crypt refresh_token, because Kestrel has restarted? Or I am wrong?

I also check OpenIddict tables and OpenIddictApplications, OpenIddictAuthorizations and OpenIddictScopes are all empty. Only OpenIddictTokens contain some data (and all are Type refresh_token):

I would expect, that refresh_tokens are saved somewhere. Where? Maybe this is the source problem, why are my refresh_tokens invalid after some random time (maybe when Kestrel is restarted).

IIS log:

Hosting environment: Production
Content root path: D:\Podatki\OpPISWeb\WWWProduction
Now listening on: http://localhost:1408
Application started. Press Ctrl+C to shut down.
fail: AspNet.Security.OpenIdConnect.Server.OpenIdConnectServerMiddleware[0]
      The token request was rejected because the authorization code or the refresh token was invalid.
fail: AspNet.Security.OpenIdConnect.Server.OpenIdConnectServerMiddleware[0]
      The token request was rejected because the authorization code or the refresh token was invalid.

Here are my Startup.cs:

public void ConfigureServices(IServiceCollection services)
{
    try
    {
        services.Configure<IISOptions>(options =>
        {
        });

        services.AddMvc();
        services.AddMvcCore().AddDataAnnotations();

        services.AddEntityFrameworkSqlServer();

        services.AddScoped<UserStore<AppUser, AppRole, AppDbContext, int, AppUserClaim, AppUserRole, AppUserLogin, AppUserToken, AppRoleClaim>, AppUserStore>();
        services.AddScoped<UserManager<AppUser>, AppUserManager>();
        services.AddScoped<RoleManager<AppRole>, AppRoleManager>();
        services.AddScoped<SignInManager<AppUser>, AppSignInManager>();
        services.AddScoped<RoleStore<AppRole, AppDbContext, int, AppUserRole, AppRoleClaim>, AppRoleStore>();

        var connection = Configuration["ConnectionStrings:Web"];
        services.AddDbContext<AppDbContext>(options =>
        {
            options.UseSqlServer(connection);
            options.UseOpenIddict<int>();
            if (this.env.IsDevelopment())
                options.EnableSensitiveDataLogging();
        });


        services
            .AddIdentity<AppUser, AppRole>()
            .AddUserStore<AppUserStore>()
            .AddUserManager<AppUserManager>()
            .AddRoleStore<AppRoleStore>()
            .AddRoleManager<AppRoleManager>()
            .AddSignInManager<AppSignInManager>()
            .AddDefaultTokenProviders();

        services.Configure<IdentityOptions>(options =>
            {
                options.ClaimsIdentity.UserNameClaimType = OpenIdConnectConstants.Claims.Name;
                options.ClaimsIdentity.UserIdClaimType = OpenIdConnectConstants.Claims.Subject;
                options.ClaimsIdentity.RoleClaimType = OpenIdConnectConstants.Claims.Role;
            });

        services.AddOpenIddict<int>(options =>
        {
            options.AddEntityFrameworkCoreStores<AppDbContext>();
            options.AddMvcBinders();
            options.EnableTokenEndpoint("/API/authorization/token");
            options.AllowPasswordFlow();
            options.AllowRefreshTokenFlow();
            options.AllowCustomFlow("urn:ietf:params:oauth:grant-type:google_identity_token");
            options.AllowCustomFlow("urn:ietf:params:oauth:grant-type:logedin");
            options.UseJsonWebTokens();
            if (this.env.IsDevelopment())
                options.AddEphemeralSigningKey();  
            else
                options.AddSigningCertificate(new FileStream(
                    Directory.GetCurrentDirectory() + "/Resources/cert.pfx", FileMode.Open), "password");
            options.SetAccessTokenLifetime(TimeSpan.FromMinutes(30));
            options.SetRefreshTokenLifetime(TimeSpan.FromDays(14));
            if (this.env.IsDevelopment())
                options.DisableHttpsRequirement();
        });

        services.AddSingleton<DbSeeder>();
        services.AddSingleton<IConfiguration>(c => { return Configuration; });

    }
    catch (Exception ex)
    {
        Console.WriteLine(ex.ToString());
        throw;
    }
}

public void Configure(IApplicationBuilder app, IHostingEnvironment env, ILoggerFactory loggerFactory, DbSeeder dbSeeder)
{
    loggerFactory.AddConsole(this.Configuration.GetSection("Logging"));
    loggerFactory.AddDebug();

    if (env.IsDevelopment())
    {
        app.UseDeveloperExceptionPage();
        app.UseWebpackDevMiddleware(new WebpackDevMiddlewareOptions
        {
            HotModuleReplacement = true
        });
    }
    app.UseStaticFiles();

    app.UseStaticFiles(new StaticFileOptions()
    {
        FileProvider = new PhysicalFileProvider(this.Configuration["Directories:Upload"]),
        RequestPath = new PathString("/Files")
    });

    app.UseOpenIddict();

    var JwtOptions = new JwtBearerOptions()
    {
        Authority = this.Configuration["Authentication:OpenIddict:Authority"],
        Audience = "OpPISWeb",
        AutomaticAuthenticate = true,
        AutomaticChallenge = true,

        RequireHttpsMetadata = false
    };
    JwtOptions.RequireHttpsMetadata = !env.IsDevelopment();
    app.UseJwtBearerAuthentication(JwtOptions);

    app.UseMvc();

    using (var context = new AppDbContext(this.Configuration))
    {
        context.Database.Migrate();
    }
    try
    {
        dbSeeder.SeedAsync();
    }
    catch (AggregateException e)
    {
        throw new Exception(e.ToString());
    }
}

Console screenshots:

Update:
In the end all I had to do is:

services.AddDataProtection()
                    .SetApplicationName(this.Configuration["Authentication:ApplicationId"])
                    .PersistKeysToFileSystem(new DirectoryInfo(this.Configuration["Directories:Keys"]));

Don't forget to add rights to IIS for Directories:Keys folder.

Answer 1

I would expect, that refresh_tokens are saved somewhere. Where?

Nowhere. The authorization codes, refresh tokens and access tokens (when using the default format) issued by OpenIddict are self-contained and are never stored for security reasons (only metadata like the subject or the authorization identifier associated with the token is).

The issue you're seeing is likely caused by the fact you haven't configured your environment to correctly persist the cryptographic keys used by the ASP.NET Core Data Protection stack OpenIddict relies on to encrypt its tokens. You can read OpenIddict: 401 errors when two or more service instance count for more information on how to fix that.