How to update PDF file using itext while it is open with PdfiumViewer in c#?

Question

I have a small Windows Forms application. It is supposed to open a PDF file, then put a stamp on it. Afterward, the updated document should be display for the user to see the change.

However, I am having some troubles updating the PDF.

I am using PdfiumViewer.PdfRenderer to display the document:

    private void LoadPdf(string pdfPath)
    {
        pdfDocument = PdfiumViewer.PdfDocument.Load(pdfPath);
        pdfRenderer.Load(pdfDocument);            
    }

Then I am using itext to put a stamp on it:

        using (PdfReader pdfReader = new PdfReader(pdfPath))
        using (PdfWriter pdfWriter = new PdfWriter(pdfPath))
        using (iText.Kernel.Pdf.PdfDocument pdfDoc = new iText.Kernel.Pdf.PdfDocument(pdfReader, pdfWriter))
        {
            pdfWriter.SetCloseStream(true);
            Document document = new Document(pdfDoc);

            // Add a stamp at the captured position
            Paragraph stamp = new Paragraph("bababa")
                .SetFixedPosition(1, pdfX, pdfY, 50)
                .SetFontSize(12)
                .SetBackgroundColor(iText.Kernel.Colors.ColorConstants.YELLOW);

            document.Add(stamp);
            
        }
        LoadPdf(pdfPath);

The "PDFWriter" gives me an error, that file is used by another process (I guess PdfiumViewer.PdfRenderer locks it). So I wanted to ask how can I "unlock" the file for the stamping process.

I know that I can kind of achieve this by saving the stamped document as another file and then opening it, but that resets the display of the PDF on Renderer and the user has to scroll/zoom back again. That is why I want to do it using 1 file.

If it is not possible to do it using 1 file, what are the best practices for such app?

Edit:

Tried saving file to MemoryStream first as per @mkl's suggestion, but without sucess:

        MemoryStream memStream = new MemoryStream();           
        using (PdfReader pdfReader = new PdfReader(pdfPath))
        using (PdfWriter pdfWriter = new PdfWriter(memStream))
        using (iText.Kernel.Pdf.PdfDocument pdfDoc = new iText.Kernel.Pdf.PdfDocument(pdfReader, pdfWriter))
        {
            pdfWriter.SetCloseStream(true);
            Document document = new Document(pdfDoc);

            // Add a stamp at the captured position
            Paragraph stamp = new Paragraph("bababa")
                .SetFixedPosition(1, pdfX, pdfY, 50)
                .SetFontSize(12)
                .SetBackgroundColor(iText.Kernel.Colors.ColorConstants.YELLOW);

            document.Add(stamp);
            
        }
        using (FileStream fileStream = new FileStream(pdfPath, FileMode.Create, FileAccess.Write))
        {
            memStream.Position = 0; // Reset the position to the beginning of the stream
            memStream.CopyTo(fileStream);
        }

Still the same error, but now when trying to write MemoryStream to PDF