How can I prevent my PKCanvas PDFOverlay object from becoming blurry when pinch-zooming in on a PDF in iOS?

Question

I got the code presented in WWDC2021 to work and im able to overlay a PencilKit Canvas object over each pdf page.

When starting the app, a pdf is shown and the overlayed PKCanvas looks as expected. When pinch-zooming in on the pdf, the PKCanvas gets pixelated and blurry.

the zoomed in pdf with the pixelated stroke

How the overlay is created:

class MyPDFPage : PDFPage {
    var drawing: PKDrawing?
}

extension ReaderView {
    class Coordinator: NSObject, PDFPageOverlayViewProvider, PDFViewDelegate {
        var pageToViewMapping = [PDFPage: PKCanvasView]()

        func pdfView(_ view: PDFView, overlayViewFor page: PDFPage) -> UIView? {
            var resultView: PKCanvasView? = nil
            if let overlayView = pageToViewMapping[page] {
                resultView = overlayView
            } else {
                
                let canvasView = PKCanvasView()
                
                canvasView.backgroundColor = .clear
                canvasView.drawingPolicy = .pencilOnly
                
                canvasView.tool = PKInkingTool(.pen, color: .black, width: 1)
                
                resultView = canvasView
                
                pageToViewMapping[page] = resultView
            }
            
            if let page = page as? MyPDFPage, let drawing = page.drawing {
                resultView?.drawing = drawing
            }
                        
            return resultView
        }
    }
}

How the pdf view is created:

struct ReaderView: UIViewRepresentable {
    
    public typealias UIViewType = PDFView
    let url: URL
    let document: PDFDocument
    let pdfView = PDFView()
    
    init(_ url: URL) {
        self.url = url
        self.document = PDFDocument(url: self.url)!
    }
    
    func makeCoordinator() -> Coordinator {
        Coordinator()
    }
    
    func makeUIView(context: Context) -> PDFView {
        pdfView.displayDirection = .vertical
        pdfView.isInMarkupMode = true
        pdfView.pageOverlayViewProvider = context.coordinator
        pdfView.displayMode = .singlePageContinuous
        pdfView.usePageViewController(false)
        pdfView.backgroundColor = .clear
        pdfView.delegate = context.coordinator
        
        pdfView.document = document
        return pdfView
    }
    
    func updateUIView(_ uiView: PDFView, context: Context) {
        print(pdfView.scaleFactor)
    }
}

I have tried to use canvasView.maximumZoomScale = 10 which allows me to zoom into the drawing.

When using this, the overlay can be correctly zoomed in but then the PDF underneath does not react to the pinch-input. This means the overlay is scaled on its own with an anchor point on the top-left side of the page.

setting maximumZoomScale to 10 allows to zoom in only on the PKCanvas but the pdf ignores the input