radio23
radio23

Reputation: 99

What is holding the GIL and how can it be released


from concurrent.futures import ThreadPoolExecutor
import time
import io
import pandas as pd
from image_exporter import export
import numpy as np

df1 = pd.DataFrame(np.random.rand (6, 4))
df2 = pd.DataFrame(np.random.rand (6, 4))
df3 = pd.DataFrame(np.random.rand (6, 4))
df4 = pd.DataFrame(np.random.rand (6, 4))

def get_exported(df):
    s = time.time()
    buffer = io.BytesIO()
    export(df.style.background_gradient(), buffer)
    buffer.seek(0)
    print('get_exported', time.time() - s)
    
def test_parallel():
    tasks = [df1, df2, df3, df4]
    with ThreadPoolExecutor() as executor:
        executor.map(get_exported, tasks)

def test_seq():
    tasks = [df1, df2, df3, df4]
    for t in tasks:
        get_exported(t)
        
s = time.time()
test_parallel()
print('Total time parallel', time.time() - s)

s = time.time()
test_seq()
print('Total time sequential', time.time() - s)

image_exporter file that contains the export method https://gist.github.com/vukrado/bd14192984c51569f6ed94bcd49f4b24

The test_seq on my machine takes ~0.035 seconds per call to get_exported for a total run time of ~.14 seconds

The test_parallel on my machine takes ~0.18 seconds per call to get_exported for a total run time of ~.19 seconds

Upvotes: 0

Views: 45

Answers (0)

Related Questions