Problem with resizeObserver, React and probably re-render issue (not yet identified)

Question

I've got a problem in my application. I've made a component that wrap a children and display a "Show more/less" around it. It accepts an initialHeight and a MaxHeight which are 2 numbers, that I print in inline style. I then compare the children height and the initialHeight because if the children is smaller than I allow, there is no need to display the Show More things.

My problem is that I'm trying to handle a specific case : in one case the children is a component that contains an API call and display a loader by default. When API call is successful, I receive images and I hide the loader. Which means for the first seconds my children height is around 80px (loader) then it can be something like 600px or more (images). So I used an observer, thinking it was the best solution :

import {useState, useRef, useEffect, MutableRefObject} from 'react'
import {IoIosArrowUp, IoIosArrowDown} from 'react-icons/io'
import styles from './ShowMoreWrapper.module.scss'

type PropsTypes = {
  children: React.ReactNode
  initialHeight: number
  maxHeight: number
}

const ShowMoreWrapper = ({
  children,
  initialHeight,
  maxHeight,
}: PropsTypes): JSX.Element => {
  const [innerExpand, setInnerExpand] = useState(false)
  const [showActions, setShowActions] = useState(false)
  const childrenRef = useRef() as MutableRefObject<HTMLDivElement>
  const wrapperRef = useRef<HTMLDivElement>(null)

  const observer = new ResizeObserver(entries => {
    if (entries[0].target.clientHeight > initialHeight) {
      setShowActions(true)
      setInnerExpand(false)
    }
    observer.unobserve
  })

  useEffect(() => {
    observer.observe(childrenRef.current)
  }, [childrenRef])

  const toggleExpand = () => {
    setInnerExpand(!innerExpand)
    if (wrapperRef.current && wrapperRef.current.scrollTop > 0) {
      wrapperRef.current.scrollTop = 0
    }
  }

  return (
    <>
      <div
        ref={wrapperRef}
        className={styles.contentWrapper}
        data-inner-expand={innerExpand}
        style={{
          maxHeight: innerExpand ? `${maxHeight}px` : `${initialHeight}px`,
        }}
      >
        <div ref={childrenRef}>{children}</div>
      </div>
      {showActions && (
        <div
          className={styles.innerExpandActions}
          onClick={toggleExpand}
          data-inner-expand={innerExpand}
        >
          <span>Show {innerExpand ? 'less' : 'more'}</span>
          {innerExpand ? <IoIosArrowUp /> : <IoIosArrowDown />}
        </div>
      )}
    </>
  )
}

export default ShowMoreWrapper

I then have some CSS to handle scollbar and overflow :

.contentWrapper {
  overflow: hidden;
  transition: max-height 0.3s ease-in-out;

  &[data-inner-expand='true'] {
    overflow: auto;
  }
}

My problem is when I click on the Show more button, it tries to set innerExpand to !innerExpand to toggle but immediately it reset itself to false, probably because observer is re-defined on re-render maybe ?