issue with socket.io on nodejs with kubernetes and microservices

Question

long time i didn't had to write a post here. I guess i am really stuck... I built long time ago a monolithic app based on react and express that was handling a chat with socket.io. I remember i did struggle a little but at the end i make it work.

I am now reconverting the same app into microservices with kubernetes (on GKE) and after build the chat backend and the front, i just cannot make the chat work. It's seems somehow the socket.io instances are not connected. I tryed a lot of different things and i am now asking help regarding it. I will share bellow the parts of the code that are implying it.

CHAT BACKEND WITH EXPRESS:

There i am declaring a middleware to pass io as req.io to be able to use in a specific endpoint. This part work fine (at least it's seems to me)

require("express-async-errors")
const express = require("express")
const helmet = require("helmet")
const socket = require("socket.io")
const http = require("http")
const compression = require("compression")
const bodyParser = require("body-parser")
const cookieSession = require("cookie-session")
const cookieParser = require("cookie-parser")

// IMPORT ROUTES
const routes = require("./routes")

// IMPORT MIDDLWARES
const Import = require("@archsplace/archsplace_commun")
const isError = Import("middlewares", "isError")
const isCurrentUser = Import("middlewares", "isCurrentUser")
const { NotFoundError } = Import("factory", "errors")

// LAUNCH EXPRESS
const app = express()
const server = http.createServer(app)
const io = socket(server)

const secure = process.env.NODE_ENV !== "test"

// USE MAIN MIDDLWWARE
app.set("trust proxy", true)
app.use(helmet())
app.disable("x-powered-by")
app.use(bodyParser.urlencoded({ extended: false }))
app.use(bodyParser.json())
app.use(cookieSession({ signed: false, secure }))
app.use(isCurrentUser)
app.use(cookieParser())
app.use(compression())
app.use((req, res, next) => {
  req.io = io
  next()
})

// USE ROUTES
routes.map(route => app.use(route.url, route.path))
app.all("*", async (req, res) => {
  throw new NotFoundError()
})

// USE CUSTOM MIDDLWWARE
app.use(isError)

module.exports = server

Then i have a endpoint to emit a chat event, its the same endpoint i use to POST my message in my mongoDB database. I think this part work somehow, but not sure where it's emitting exactly

ENDPOINT TO EMIT SOCKET EVENT

const express = require("express")
const db = require("mongoose")

// DATABASE AND LIBRARIES
const Import = require("@archsplace/archsplace_commun")
const isAuthenticated = Import("middlewares", "isAuthenticated")
const isActivated = Import("middlewares", "isActivated")
const Chat = Import("models", "Chat", "chat")
const Room = Import("models", "ChatRoom", "chat")

// EVENTS
const { NatsWrapper } = require("../../../services/natsWrapper")
const { RoomUpdatedPub } = require("../../../events/publishers/roomUpdatedPub")
const { ChatCreatedPub } = require("../../../events/publishers/chatCreatedPub")

// VALIDATES
const { BadRequestError, DatabaseConnectionError } = Import("factory", "errors")

const router = express.Router()

// @route  POST api/chat/private/message/:roomId
// @desc   Post Chat message by chatroom id
// @access Private
router.post("/:roomId", isAuthenticated, isActivated, async (req, res) => {
  // DEFINE QUERIES
  let chat
  const { message, avatar } = req.body
  const { roomId } = req.params

  // ENSURE ROOM EXIST FOR USER
  const room = await Room.findOne({ $and: [{ _id: roomId }, { _users: { $elemMatch: { _user: req.user._id } } }] })
  if (!room) {
    throw new BadRequestError("This chatroom doesn't exist")
  }

  // CREATE CHAT
  const chatFields = {
    message,
    _emitter: req.user._id,
    _chatId: roomId
  }

  // EMIT TO SOCKET
  req.io.emit(roomId, {
    ...chatFields,
    avatar: avatar,
    read: 1,
    role: req.user.authorities
  })

  // HANDLE MONGODB TRANSACTIONS
  const SESSION = await db.startSession()
  try {
    // CREATE CHAT
    await SESSION.startTransaction()
    chat = await new Chat(chatFields).save()
    await room.set({ lastUpdated: Date.now() }).save()
    await new RoomUpdatedPub(NatsWrapper.client()).publish(room)
    await new ChatCreatedPub(NatsWrapper.client()).publish(chat)
    await SESSION.commitTransaction()

    // RETURN AND FINALIZE ENDPOINT
    res.status(201).json(chat)
  } catch (e) {
    // CATCH ANY ERROR DUE TO TRANSACTION
    await SESSION.abortTransaction()
    console.error(e)
    throw new DatabaseConnectionError()
  } finally {
    // FINALIZE SESSION
    SESSION.endSession()
  }
})

module.exports = router

INGRESS NGINX CONGIS

This probably could be wrong, i saw in internet that we need to use this annotation nginx.ingress.kubernetes.io/websocket-services and i am attributing to the server where i build the chat (and where i am using socket.io)

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: ingress-service
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/use-regex: "true"
    nginx.ingress.kubernetes.io/permanent-redirect-code: "301"
    nginx.ingress.kubernetes.io/from-to-www-redirect: "true"
    # SOCKET CONFIGURATIONS
    nginx.ingress.kubernetes.io/websocket-services: "chat-srv"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "1800"
    nginx.ingress.kubernetes.io/proxy-read-timeout: "1800"
spec:
  rules:
    - host: www.archsplace.dev
      http:
        paths:
          - path: /api/architect/?(.*)
            backend:
              serviceName: architect-srv
              servicePort: 3000
          - path: /?(.*)
            backend:
              serviceName: client-website-srv
              servicePort: 3000
    - host: architects.archsplace.dev
      http:
        paths:
          - path: /auth/?(.*)
            backend:
              serviceName: auth-srv
              servicePort: 3000
          - path: /api/account/?(.*)
            backend:
              serviceName: account-srv
              servicePort: 3000
          - path: /api/architect/?(.*)
            backend:
              serviceName: architect-srv
              servicePort: 3000
          - path: /api/chat/?(.*)
            backend:
              serviceName: chat-srv
              servicePort: 3000
          - path: /?(.*)
            backend:
              serviceName: client-architects-srv
              servicePort: 3000
    - host: users.archsplace.dev
      http:
        paths:
          - path: /auth/?(.*)
            backend:
              serviceName: auth-srv
              servicePort: 3000
          - path: /api/account/?(.*)
            backend:
              serviceName: account-srv
              servicePort: 3000
          - path: /api/architect/?(.*)
            backend:
              serviceName: architect-srv
              servicePort: 3000
          - path: /api/chat/?(.*)
            backend:
              serviceName: chat-srv
              servicePort: 3000
          - path: /?(.*)
            backend:
              serviceName: client-users-srv
              servicePort: 3000
    - host: partners.archsplace.dev
      http:
        paths:
          - path: /auth/?(.*)
            backend:
              serviceName: auth-srv
              servicePort: 3000
          - path: /api/account/?(.*)
            backend:
              serviceName: account-srv
              servicePort: 3000
          - path: /api/chat/?(.*)
            backend:
              serviceName: chat-srv
              servicePort: 3000
          - path: /?(.*)
            backend:
              serviceName: client-partners-srv
              servicePort: 3000
    - host: admin.archsplace.dev
      http:
        paths:
          - path: /auth/?(.*)
            backend:
              serviceName: auth-srv
              servicePort: 3000
          - path: /api/account/?(.*)
            backend:
              serviceName: account-srv
              servicePort: 3000
          - path: /api/architect/?(.*)
            backend:
              serviceName: architect-srv
              servicePort: 3000
          - path: /api/chat/?(.*)
            backend:
              serviceName: chat-srv
              servicePort: 3000
          - path: /?(.*)
            backend:
              serviceName: client-admin-srv
              servicePort: 3000
    - host: business.archsplace.dev
      http:
        paths:
          - path: /?(.*)
            backend:
              serviceName: client-business-srv
              servicePort: 3000

Then in the client side i am using the library socket.io-client on my react app.

CLIENT UTIL WHERE I DECLARE THE IO LIBRARY

There since i use architects.archsplace.dev, i assume that the chat server would be only available at /api/chat as it's defined on ingress nginx but i am not sure..

import io from "socket.io-client"

export const socket = io(`${window.location.host}/api/chat`, {
  reconnect: true
})

Then i build a UI with the chat where i am actually trying to receive the chat info and store it into a react state:

REACT COMPONENT WITH USEEFFECT TO LOAD SOCKETS

Here also you can see in the use effect i am using the roomId to connect to the same event i emit in the backend part previously. (this was working on a monolithic but not here)

import React, { useState, useEffect } from "react"
import { connect } from "react-redux"
import Timestamp from "react-timestamp"

// IMPORT ACTIONS
import { sendMessage, getMessages } from "@actions/chatActions"

// IMPORT COMPONENTS
import ChatMessage from "./ChatMessage"

// IMPORT UTILS
import { socket, isEmpty, imageRender } from "@utils"

const ChatRoom = ({ sendMessage, getMessages, classes, room, user, chat: { messages } }) => {
  // HOOKS
  const [state, setState] = useState({
    message: "",
    messages: [],
    errors: {},
    typing: false,
    trigger: false,
    isInteracted: false,
    page: 0,
    limit: 20
  })
  const target = room.users.find(({ _id }) => _id !== user._id)
  const isOnline = Math.floor(Date.now() - new Date(target.lastConnectionDate).getTime() / 1000) && target.isOnline

  // USE EFFECT
  useEffect(() => {
    getMessages(state.page, state.limit, room._id)
  }, [getMessages, state.page, state.limit, room._id])

  useEffect(() => {
    const handleMessageSocket = () => {
      const addMessage = data => setState(prevStates => ({ ...prevStates, messages: [...state.messages, data] }))
      socket.on(room._id, data => addMessage(data))
    }
    const clearMessageSocket = () => {
      socket.off(room._id)
      setState(prevStates => ({ ...prevStates, messages: [] }))
    }
    // LOAD DATA FROM REDUCER
    setState(prevStates => ({ ...prevStates, messages }))
    // LOAD SOCKETS
    handleMessageSocket()
    return () => clearMessageSocket()
  }, [room._id, state.messages, messages])

  // HANDLE FUNCTIONS
  const handleMessage = e => setState(prevStates => ({ ...prevStates, message: e.target.value }))
  const clearMessage = () => setState(prevStates => ({ ...prevStates, message: "" }))
  const handleSubmit = e => {
    e.preventDefault()
    const chatMessage = { message: state.message, avatar: imageRender(user.avatar, "tr:n-user_avatar_small") }
    !isEmpty(state.message) && sendMessage(chatMessage, room._id)
    clearMessage()
  }
  // RENDER CHATROOM ITEM
  const renderAvatar = () => {
    return (
      <div className={classes.chatRoomItemAvatar}>
        <img src={imageRender(target.avatar, "tr:n-user_avatar_small")} alt={target.name} />
        {isOnline && <div className={classes.chatRoomItemActive} />}
      </div>
    )
  }
  const renderInfo = () => {
    return (
      <div className={classes.chatRoomItemInfo}>
        <h3>{target.name}</h3>
        <p>
          <i>access_time</i>
          <Timestamp className="request-item-timestamp" relative date={target.lastConnectionDate} autoUpdate />
        </p>
      </div>
    )
  }
  const renderChatItem = () => {
    return (
      <div className={classes.chatRoomItem}>
        {renderAvatar()}
        {renderInfo()}
      </div>
    )
  }
  // RENDER MESSAGE AREA
  const renderMessages = () => {
    return <div className={classes.chatMessages}>{JSON.stringify(state.messages.map(i => i.message))}</div>
  }

  // RENDER INPUT AREA
  const renderInput = () => {
    return (
      <form className={classes.chatInputWrapper} autoComplete="off" onSubmit={e => handleSubmit(e)}>
        <div className={classes.chatInput}>
          <input type="text" placeholder="Message" value={state.message} onChange={handleMessage} />
          <button type="submit">
            <i>reply</i>
          </button>
        </div>
      </form>
    )
  }

  // MAIN
  return (
    <div className={classes.chatRoom}>
      {renderChatItem()}
      {renderMessages()}
      {renderInput()}
    </div>
  )
}

const mapStateToProps = state => ({
  chat: state.chat
})

export default connect(mapStateToProps, { sendMessage, getMessages })(ChatRoom)

So if anyone have been confronted to the same issue, and know what i might have done wrong, i am completly stuck. I even tried to setup a redis service and pass the socket throught a redis io adapter but didn't work either...

Answer 1

I found a solution there for the people that might struggle like me...i think it's a bit of hacky but it work well.

I was observing in my front that the socket were all the time triggering under the /socket.io/.... and if you take a look at my ingress nginx that would look into my react app and return a 404 page probably.

So i forced my chat-srv to be present there on this specific endpoint with the following code:

 - path: /socket.io/?(.*)
   backend:
     serviceName: chat-srv
     servicePort: 3000

Then once i did this, it was actually looking into my chat backend and worked out. I need also to specify that i use the version 2.2.0 of socket.io

I tried with the version 3 but it didn't work.