ASP.Net ASOS JWT return Token but doesn't authorize

Question

I'm building a simple application with a fixed credential, so there's no need to integrate with EF or whatever.

I'm trying to use the AspNet.Security.OpenIdConnect.Server package with JWT configuration, I can validate my credentials and return the Token, but using [Authorize] attribute my endpoint always return 401 unauthorized.

Startup.cs

services.AddAuthentication(options =>
{
    options.DefaultAuthenticateScheme = JwtBearerDefaults.AuthenticationScheme;
    options.DefaultChallengeScheme = JwtBearerDefaults.AuthenticationScheme;
})
.AddOpenIdConnectServer(options =>
{
    options.AllowInsecureHttp = true;
    options.AccessTokenLifetime = TimeSpan.FromHours(1);
    options.TokenEndpointPath = "/v1/authtoken";
    options.AccessTokenHandler = new JwtSecurityTokenHandler
    {
        OutboundClaimTypeMap = new Dictionary<string, string>()
    };
    options.Provider = new AuthorizationProvider();
    options.SigningCredentials.AddKey(Key);
})
.AddJwtBearer();

....

app.UseAuthentication();

AuthorizationProvider.cs

public override Task HandleTokenRequest(HandleTokenRequestContext context)
{
    if (context.Request.IsClientCredentialsGrantType())
    {
        if (!string.Equals(context.Request.ClientId, "username", StringComparison.Ordinal) ||
            !string.Equals(context.Request.ClientSecret, "password, StringComparison.Ordinal))
        {
            context.Reject(
                error: OpenIdConnectConstants.Errors.InvalidGrant,
                description: "Invalid user credentials.");

            return Task.CompletedTask;
        }

        var identity = new ClaimsIdentity(context.Scheme.Name);

        identity.AddClaim(new Claim(OpenIdConnectConstants.Claims.Subject, "subject"));
        identity.AddClaim(ClaimTypes.Role, "user", OpenIdConnectConstants.Destinations.AccessToken,
            OpenIdConnectConstants.Destinations.AccessToken);

        var ticket = new AuthenticationTicket(
            new ClaimsPrincipal(identity),
            new AuthenticationProperties(), context.Scheme.Name);
        ticket.SetResources("resource_server");

        context.Validate(ticket);
    }

    return Task.CompletedTask;
}

Controller

[Authorize(Roles = "user")]
public class ValuesController : Controller

According to jwt.io, this is the payload inside the JWT:

{
  "sub": "subject",
  "http://schemas.microsoft.com/ws/2008/06/identity/claims/role": "user",
  "token_usage": "access_token",
  "jti": "7b5d3180-b535-489b-97ec-fccaedf9615a",
  "cfd_lvl": "private",
  "aud": "resource_server",
  "azp": "username",
  "nbf": 1550510623,
  "exp": 1550514223,
  "iat": 1550510623,
  "iss": "https://localhost:44302/"
}

Also tried to use just [Authorize], but I also get 401 unauthorized.

If I use the default ASOS Token, it work without any problem, just having problems when using JWT (Which I'm trying to use because I want to add a Custom Key and not a Certificate).

Postman Request body