E*Trade API frequently returns HTTP 401 Unauthorized when fetching an access token but not always

Question

Summary

I have written a simple C# .NET Core application to authenticate against the E*Trade API using OAuthv1 with the intention of fetching stock quotes. I am able to authenticate and get a request token, redirect to the authorization page and obtain a verifier string. However, when I use the verifier string to perform the access token request, roughly 9 times out of 10 I get 401 unauthorized. But then occasionally it works and I get the access token back.

Details

• I'm using the .NET OAuth class OAuthRequest to generate the query string authorization params.
• I am using this API https://apisb.etrade.com/docs/api/authorization/get_access_token.html#
• I have downloaded this sample application and compared the URLs being used and have found no major discrepancy that would explain this behavior. https://cdn2.etrade.net/1/18122609420.0/aempros/content/dam/etrade/developer-site/en_US/document/downloads/EtradePythonClient.zip
• The sample application works every time with my creds, so I know they are functional. There is some difference in the way the C# code generates the signature (maybe) that is causing this issue and it is clearly nondeterministic because sometimes my app works.
• I have compared the URLs being used for auth between the sample app and mine and they are the same.

Code

I have created separate request objects for the sake of sanity, I won't leave it this way. Again, I'm able to fetch the request tokens, redirect to authorize and get the verifier string, just not the access token.

    private static async Task FetchData()
    {  
        // Values
        string consumerKey = "...";
        string consumerSecret = "...";
        string requestTokenUrl = "https://api.etrade.com/oauth/request_token";
        string authorizeUrl = "https://us.etrade.com/e/t/etws/authorize";
        string accessTokenUrl = "https://api.etrade.com/oauth/access_token";
        string quoteUrl = "https://api.etrade.com/v1/market/quote/NVDA,DJI";

        // Create the request 
        var request = new OAuthRequest
        {
            Type = OAuthRequestType.RequestToken,
            ConsumerKey = consumerKey,
            ConsumerSecret = consumerSecret,
            Method = "GET",
            RequestUrl = requestTokenUrl,
            Version = "1.0",
            Realm = "etrade.com",
            CallbackUrl = "oob",
            SignatureMethod = OAuthSignatureMethod.HmacSha1
        };

        // Make call to fetch session token
        try
        {
            HttpClient client = new HttpClient();
            
            var requestTokenUrlWithQuery = $"{requestTokenUrl}?{request.GetAuthorizationQuery()}";
            var responseString = await client.GetStringAsync(requestTokenUrlWithQuery);
            var tokenParser = new TokenParser(responseString, consumerKey);

            // Call authorization API
            var authorizeUrlWithQuery = $"{authorizeUrl}?{tokenParser.GetQueryString()}";
            
            // Open browser with the above URL 
            ProcessStartInfo psi = new ProcessStartInfo
            {
                FileName = authorizeUrlWithQuery,
                UseShellExecute = true
            };
            Process.Start(psi);

            // Request input of token, copied from browser
            Console.Write("Provide auth code:");
            var authCode = Console.ReadLine();
           
            // Need auth token and verifier
            var secondRequest = new OAuthRequest
            {
                Type = OAuthRequestType.AccessToken,
                ConsumerKey = consumerKey,
                ConsumerSecret = consumerSecret,
                SignatureMethod = OAuthSignatureMethod.HmacSha1,
                Method = "GET",
                Token = tokenParser.Token,
                TokenSecret = tokenParser.Secret,
                Verifier = authCode,
                RequestUrl = accessTokenUrl,
                Version = "1.0",
                Realm = "etrade.com"
            };

            // Make access token call
            var accessTokenUrlWithQuery = $"{accessTokenUrl}?{secondRequest.GetAuthorizationQuery()}";
            responseString = await client.GetStringAsync(accessTokenUrlWithQuery);

            Console.WriteLine("Access token: " + responseString);

            // Fetch quotes
            tokenParser = new TokenParser(responseString, consumerKey);
            var thirdRequest = new OAuthRequest
            {
                Type = OAuthRequestType.ProtectedResource,
                ConsumerKey = consumerKey,
                ConsumerSecret = consumerSecret,
                SignatureMethod = OAuthSignatureMethod.HmacSha1,
                Method = "GET",
                Token = tokenParser.Token,
                TokenSecret = tokenParser.Secret,
                RequestUrl = quoteUrl,
                Version = "1.0",
                Realm = "etrade.com"
            };
            
            var quoteUrlWithQueryString = $"{quoteUrl}?{thirdRequest.GetAuthorizationQuery()}";
            responseString = await client.GetStringAsync(quoteUrlWithQueryString);

            // Dump data to console 
            Console.WriteLine(responseString);
            
        }
        catch (Exception ex)
        {
            Console.WriteLine("\n"+ ex.Message);
        }
    }

    class TokenParser {
        private readonly string consumerKey;

        public TokenParser(string responseString, string consumerKey)
        {
            NameValueCollection queryStringValues = HttpUtility.ParseQueryString(responseString);
            Token = HttpUtility.UrlDecode(queryStringValues.Get("oauth_token"));
            Secret = HttpUtility.UrlDecode(queryStringValues.Get("oauth_token_secret"));
            this.consumerKey = consumerKey;
        }

        public string Token { get; set; }
        public string Secret { get; private set; }

        public string GetQueryString()
        {
            return $"key={consumerKey}&token={Token}";
        }
    }

As an example, while writing this post I ran the app a couple times and it worked once and failed once. I didn't change the code at all.

Answer 1

This was amazingly helpful. I used your code plus what was posted here to automate this (because tokens expire daily): ETrade API unattended authentication

I made two edits:

1. Changed the authorize URL to what was posted here: https://seansoper.com/blog/connecting_etrade.html

2. For the log on button, changed it to search by ID: Button btnLogOn = StaticInstanceHelper.Browser.Button(Find.ById("logon_button"));

I ran into issues with Watin and setting up the Apartmentstate. So did this:

    static void Main(string[] args)
    {
        System.Threading.Thread th = new Thread(new ThreadStart(TestAuth));
        th.SetApartmentState(ApartmentState.STA);
        th.Start();
        th.Join();
    }

Then put your code in the TestAuth method.

Answer 2

I ran into a similar problem (although I'm using JavaScript).

The Get Request Token call (/request_token) call would work, and I could successfully open the Authorize Application page in a web browser, where the user could successfully authorize and receive the oauth_verifier token. However, when I tried to sign a Get Access Token request, I would receive a 401 - oauth_problem=signature_invalid.

The reason turned out to be that the oauth_signature and other parameters must be percent-encoded (rfc3986). In the case of the Authorize Application flow, we are lucky that the web browser will automatically percent-encode parameters in URL bar. However, for the Get Access Token call, this does not involve the web browser, so URL parameters were not getting percent encoded.

For example, instead of oauth_signature equal to abc123=, we need oauth_signature equal to abc123%3D.

This can be fixed by rfc3986-encoding the parameters in HTTP requests.

The reason it worked 1 times out of 10 is probably because you got lucky that the parameters did not contain any characters which needed to be rfc3986-encoded.

Answer 3

As a sanity check I plugged my auth params into a site that would generate the signature just to see if it was the same as what I was getting out of OAuthRequest. It was not. I decided to try something different. I implemented my logic using RestSharp and got it working almost immediately. Here is the code.

// Values
        string consumerKey = "...";
        string consumerSecret = "...";
        string baseEtradeApiUrl = "https://api.etrade.com";
        string baseSandboxEtradeApiUrl = "https://apisb.etrade.com";
        string authorizeUrl = "https://us.etrade.com";  
        
        try
        {
            // Step 1: fetch the request token
            var client = new RestClient(baseEtradeApiUrl);
            client.Authenticator = OAuth1Authenticator.ForRequestToken(consumerKey, consumerSecret, "oob");
            IRestRequest request = new RestRequest("oauth/request_token");
            var response = client.Execute(request);
            Console.WriteLine("Request tokens: " + response.Content);

            // Step 1.a: parse response 
            var qs = HttpUtility.ParseQueryString(response.Content);
            var oauthRequestToken = qs["oauth_token"];
            var oauthRequestTokenSecret = qs["oauth_token_secret"];

            // Step 2: direct to authorization page
            var authorizeClient = new RestClient(authorizeUrl);
            var authorizeRequest = new RestRequest("e/t/etws/authorize");
            authorizeRequest.AddParameter("key", consumerKey);
            authorizeRequest.AddParameter("token", oauthRequestToken);
            ProcessStartInfo psi = new ProcessStartInfo
            {
                FileName = authorizeClient.BuildUri(authorizeRequest).ToString(),
                UseShellExecute = true
            };
            Process.Start(psi);

            Console.Write("Provide auth code:");
            var verifier = Console.ReadLine();

            // Step 3: fetch access token
            var accessTokenRequest = new RestRequest("oauth/access_token");
            client.Authenticator = OAuth1Authenticator.ForAccessToken(consumerKey, consumerSecret, oauthRequestToken, oauthRequestTokenSecret, verifier);
            response = client.Execute(accessTokenRequest);
            Console.WriteLine("Access tokens: " + response.Content);

            // Step 3.a: parse response 
            qs = HttpUtility.ParseQueryString(response.Content);
            var oauthAccessToken = qs["oauth_token"];
            var oauthAccessTokenSecret = qs["oauth_token_secret"];

            // Step 4: fetch quote
            var sandboxClient = new RestClient(baseSandboxEtradeApiUrl);
            var quoteRequest = new RestRequest("v1/market/quote/GOOG.json");
            sandboxClient.Authenticator = OAuth1Authenticator.ForProtectedResource(consumerKey, consumerSecret, oauthAccessToken, oauthAccessTokenSecret);
            response = sandboxClient.Execute(quoteRequest);
            Console.WriteLine("Quotes: " + response.Content);

        } catch(Exception ex)
        {
            Console.WriteLine(ex.Message);
        }

The above logic works. My only working theory on the previous issue is that the signature was periodically invalid. To be honest I don't know root cause, but this solution works so I'm good with that.