Implementing scrolling on an infinite grid for Conways game of life using HTML Canvas

Question

I've set up an 'infinite' grid for Conway's Game of Life using a Set object to store only the coordinates of active cells. I've implemented scrolling by applying a fixed offset to the X and Y coordinates of active cells for horizontal and vertical scrolling, respectively. However, this approach becomes quite resource-intensive as the Set grows larger, and want to know if there is a better solution that is also not too complicated. I've included the code below. My solution is pretty simple so I'm sure there is a better way to do this, but I've only encountered solutions relating to memoization algorithms. Any suggestions on how to optimize this would be greatly appreciated.

Thank you.

const Scroll = {
    SCROLL_INCREMENT: 5,

  up: function (game, event) {
    if (event.key !== 'ArrowUp') { return; }
    this.scrollCells(game, 0, this.SCROLL_INCREMENT);
  },
  down: function (game, event) {
    if (event.key !== 'ArrowDown') { return; }
    this.scrollCells(game, 0, -this.SCROLL_INCREMENT);
  },
  left: function (game, event) {
    if (event.key !== 'ArrowLeft') { return; }
 this.scrollCells(game, this.SCROLL_INCREMENT, 0);

  },
  right: function (game, event) {
    if (event.key !== 'ArrowRight') { return; }
    this.scrollCells(game, -this.SCROLL_INCREMENT, 0);
  },
  scrollCells: function (game, dx, dy) {
    const updatedCells = new Set();
  
    for (const coordinate of game.activeCells) {
      const [x, y] = coordinate.split(",").map(Number);
      const newX = x + dx;
      const newY = y + dy;
      updatedCells.add(`${newX},${newY}`);
    }
    game.activeCells = updatedCells;
  }
};

I tried using ctx.translate() which also requires applying an offset to the set of activecells and draw them accordingly, which is not much different to my previous solution.