react-infinite-scroll-component with multiple columns

Question

I'm creating an app that shows a gallery of images (varying heights). I implemented the infinite scroll with 4 columns (masonry effect). But whenever I scroll down and the fetchMore function is called, the new images that are added to the gallery div shuffle the earlier images.

I followed a couple tutorials but the elements (images) seem to be all the same height and width so maybe my masonry effect is creating the shuffle? screenshot of old images "shuffling" due to new images

Here's my code:

const AppContext = createContext();

const AppContextProvider = ({ children }) => {
  const [allPhotos, setAllPhotos] = useState([]);
  const [pageNum, setPage] = useState(1);

  const clientID = process.env.NEXT_PUBLIC_IMG_ACCESS_KEY;

  useEffect(() => {
    fetchData();
  }, []);

  const fetchData = () => {
    axios({
      method: "GET",
      url: `https://api.unsplash.com/topics/film/photos?`,
      params: { page: pageNum, client_id: clientID, count: 10 },
    }).then((res) => {
      console.log(pageNum);
      setAllPhotos([...allPhotos, ...res.data]);
      setPage(pageNum + 1);
    });
  };

  return (
    <AppContext.Provider value={{ allPhotos, fetchData }}>
      {children}
    </AppContext.Provider>
  );
};

export { AppContextProvider, AppContext };

in my home page

import { useContext, useState } from "react";
import { AppContext } from "../components/AppContext";
import Gallery from "@/components/Gallery";
import InfiniteScroll from "react-infinite-scroll-component";
import Loader from "@/components/Loader";

export default function Home() {
  const { allPhotos, fetchData } = useContext(AppContext);

  return (
    <div className={`flex flex-col items-center p-10`}>
      <h1>Gallery Wall</h1>
      <InfiniteScroll
        dataLength={allPhotos.length} // might have to place in reducer because num is constantly changing
        next={fetchData}
        hasMore={true}
        loader={<Loader />}
        endMessage={<h4>fin.</h4>}
      >
        <Gallery images={allPhotos} />
      </InfiniteScroll>
    </div>
  );
}

In my Gallery component, I'm using tailwind css to make the columns (columns-4) and in my Image component, I just have a mb-4 (margin-bottom: 4px).

const Gallery = ({ images }) => {
  return (
    <div className={`columns-4`}>
      {images.map((img) => (
        <Image key={img.id} src={img.urls.raw} />
      ))}
    </div>
  );
};

The result I want is a smoother loading of the new images as you scroll down but not sure if that's achievable.