Syncing audio/video in pyav decoding

Question

I am making a simple video player with pyav, and using the container.decode(video=0, audio=0) method to decode both audio and video frames, then syncing them using PIL and pyaudio.

Here is the summary code/method I used:

import av
import pyaudio
from PIL import ImageTk
import tkinter as tk

...
                                    
with av.open(path, "r") as self._container:

    audio_stream = self._container.streams.audio[0]
              
    p = pyaudio.PyAudio()
    audio_stream = p.open(format=pyaudio.paFloat32, 
                          channels=audio_stream.channels, 
                          rate=audio_stream.rate, output=True)
    

    while True:
        try:
            frame = next(self._container.decode(video=0, audio=0)) # decode both audio and video simultaneously
            
            if 'Video' in repr(frame):
                # show the video frame
                self._current_img = frame.to_image() # converts the video frame to pil image  
                self._current_imgtk = ImageTk.PhotoImage(self._current_img) 
                self.label.config(image=self._current_imgtk) # display the image in a tk label

            else:
                # play the audio frame
                audio_data = frame.to_ndarray().astype('float32')
                interleaved_data = audio_data.T.flatten().tobytes()
                audio_stream.write(interleaved_data)
                
        except (StopIteration, av.error.EOFError, tk.TclError):
                break
    
    self._container = None
    audio_stream.stop_stream()
    audio_stream.close()
    p.terminate()

...

The video/audio is properly synced when decoding is done like this (according to pts):

video_frame

audio_frame 
audio_frame

video_frame

audio_frame 
audio_frame
...

But not all videos are decoded like this, instead a continues series is decoded and not synced properly:

video_frame
video_frame
video_frame
video_frame

audio_frame
audio_frame
audio_frame
audio_frame
audio_frame
audio_frame

video_frame
video_frame
video_frame
video_frame
...

it just stutters between audio and video frames

Is there any method to decode both frames evenly for all videos? Or at least provide any similar project of video player using pyav like this.

I am actually working on this project: https://github.com/Akascape/tkVideoPlayer

Thank you