Google Pubsub Error 16: Unauthenticated: Failed to retrieve auth metadata occurs when server running for a long time

Question

I have this strange issue that keeps occurring with my google pub sub service. When I start my server, the google pub sub service works and there doesn't seem to be an issue. When I leave my server running for prolonged periods of time (a day or more), once in a while, I get the following error out of nowhere.

Error: 16 UNAUTHENTICATED: Failed to retrieve auth metadata with error: request to https://www.googleapis.com/oauth2/v4/token failed, reason: getaddrinfo ENOTFOUND www.googleapis.com
    at Object.callErrorFromStatus ~\<project_path>\node_modules\@grpc\grpc-js\src\call.ts:81:24)
    at Object.onReceiveStatus (~\<project_path>\node_modules\@grpc\grpc-js\src\client.ts:338:36)
    at Object.onReceiveStatus (~\<project_path>\node_modules\@grpc\grpc-js\src\client-interceptors.ts:426:34)
    at Object.onReceiveStatus (~\<project_path>\node_modules\@grpc\grpc-js\src\client-interceptors.ts:389:48)
    at ~\<project_path>\node_modules\@grpc\grpc-js\src\call-stream.ts:276:24
    at processTicksAndRejections (node:internal/process/task_queues:78:11)
[ERROR] 08:21:29 Error: 16 UNAUTHENTICATED: Failed to retrieve auth metadata with error: request to https://www.googleapis.com/oauth2/v4/token failed, reason: getaddrinfo ENOTFOUND www.googleapis.com

If this was a credentials issue, I should see this UNAUTHENTICATED error as soon as my server starts but this is hard to diagnose because it happens randomly after a set period of time. Has anyone encountered this type of error with Google Pubsub?
I do have a generated Credentials file that you can get from your google developer console for OAuth. It has the following keys (type, project_id, private_key_id, private_key, client_email, client_id, auth_uri, token_uri, auth_provider_x509_cert_url, client_x509_cert_url).

I only use the project_id, private_key, client email. Do I need to use the others in order to stop this error from occuring? I think I linked directly to the credentials file before using an .env variable as per the google pubsub package docs and still got the same error. I know with Oauth, there is a refresh token timer of a couple of days and was wondering if that could be the cause of this. The refresh token isn't being picked up and therefore google apis cant authenticate. That's just a wild guess though. Any help would be much appreciated.

My code is below to access the service.

package.json

"@axelspringer/graphql-google-pubsub": "^2.1.0",
"googleapis": "^92.0.0",
"express": "^4.17.1",
"express-serve-static-core": "^0.1.1",
"cors": "^2.8.5",
"cookie-parser": "^1.4.5",
"apollo-server": "^3.5.0",
"apollo-server-core": "^3.5.0",
"apollo-server-express": "^3.5.0",
"graphql-tools": "^7.0.2",
"subscriptions-transport-ws": "^0.11.0",

src/index.ts

import { GooglePubSub } from "@axelspringer/graphql-google-pubsub";
const express = require("express");
const cookies = require("cookie-parser");
import cors from "cors";
import { createServer } from "http";
import { makeExecutableSchema } from "@graphql-tools/schema";
import { ApolloServer } from "apollo-server-express";
import {
  ApolloServerPluginLandingPageGraphQLPlayground,
  ApolloServerPluginLandingPageDisabled,
} from "apollo-server-core";
import {
  ConnectionContext,
  ConnectionParams,
  SubscriptionServer,
} from "subscriptions-transport-ws";

const pubSubOptions = {projectId: <GOOGLE project_id>, credentials: {client_email: <GOOGLE client_email>, private_key: <GOOGLE private_key>} }

const pubsub = new GooglePubSub(pubSubOptions);

const context = async ({ req, res, connection }: MyContext) => {
  const msgHeader = connection ? connection.context : req;
  return {
    req,
    res,
    pubsub,
  };
};

(async function () {
  const app = express();
  app.use(cors());
  const httpServer = createServer(app);

  const schema = makeExecutableSchema({
    typeDefs,
    resolvers,
  });

  const server = new ApolloServer({
    schema,
    context,
    plugins: [
      process.env.NODE_ENV === "development"
        ? ApolloServerPluginLandingPageGraphQLPlayground()
        : ApolloServerPluginLandingPageDisabled(),
      {
        async serverWillStart() {
          return {
            async drainServer() {
              subscriptionServer.close();
            },
          };
        },
      },
    ],
  });

  const subscriptionServer = SubscriptionServer.create(
    {
      schema,
      execute,
      subscribe,
      onConnect: (
        connectionParams: ConnectionParams,
        webSocket: WebSocket,
        ConnectionContext: ConnectionContext
      ) => {
        const context: any = {
          pubsub,
        };

        return context;
      },
    },
    { server: httpServer, path: server.graphqlPath }
  );

  await server.start();
  app.use(cookies());
  server.applyMiddleware({ app });

  const dbUri = db_url();

  try {
    await mongoose.connect(dbUri, configs.MONGO_OPTIONS);
  } catch (err) {
    if (err instanceof Error) {
      throw new Error(err.message);
    }
    process.exit(1);
  }

  httpServer.listen(PORT, () =>
    console.log(
      `🚀 Server ready at ${configs.apiURL}:${PORT}${server.graphqlPath}`
    )
  );
})();

Answer 1

It is probable that the error is occurring due to a token expiration, generally the Token expiration time is 1hr, but Oauth token expiration time varies depending on the token type.

You can obtain the information associated with a valid (not expired or revoked) access token using the tokeninfo API using an endpoint:

https://www.googleapis.com/oauth2/v3/tokeninfo?access_token=<access_token>

This endpoint will throw you data as:

{
  "access_type": "",
  "audience": "",
  "email": "",
  "expires_in": 0,
  "issued_to": "",
  "scope": "",
  "token_handle": "",
  "user_id": "",
  "verified_email": true
}

The important data in this is expires_in this will let you know the number of seconds until the token expires.

Adittionally you need to write the code to anticipate possibilities of expiration. In this document are the common reasons why a Token might stop working.