CODEWITHSUNDEEP
pythonpython-3.xcpu-usage
William TL
William TL

Reputation: 23

single threaded script is consuming too much cpu on a multi core processor

I have written a code to help accumulate some data over a constant time interval from a non-constant data set. It is working as expected on my current computer which is a 6 core processor (Intel i5-8400), and consumes ~25% of CPU usage. The problem is that now I am trying to rerun the same code on another computer which has 40 cores (Intel Xeon E5-2630 v4), and as soon as it starts the below function, it spikes to 55% CPU usage, while I was expecting it to be closer to 5%.

This has become an issue because I am sharing this computer with other people and it is hindering their work. Doing a comparison of the time it takes to run the script between the two computers, it seems to be negligible so I am not sure if it is actually using all of the processors (which makes sense, since I have no idea of how to multi thread loops!)

I have previously tried to find an answer, and the closest answer I could find was that it may be due to a while loop (Python script too much cpu usage) but I don't think that is applicable to my case... Most other questions were related to how to increase cpu usage, which is the opposite of my problem.

def accumulate_value (table_data_1,i_time_step):

    # table_data_1 formatted as a dataframe with columns names ['value','date_time']

    table_data_1.reset_index(drop=True,inplace=True)

    table_data_1["delta_time"] = table_data_1["date_time"].diff(periods=1).dt.seconds/60
    table_data_1["delta_time_aux"] = 0
    table_data_1["value_aux"] = 0
    table_data_1["datetime_stamp"] = pd.to_datetime(0)


    # Start loop

    for aux_counter_1 in table_data_1.index.values.tolist():

        table_data_1_aux1 = table_data_1.loc[aux_counter_1:,:]
        table_data_3 = table_data_1_aux1.loc[table_data_1_aux1.index[table_data_1_aux1.loc[:,'date_time'] - table_data_1_aux1.loc[aux_counter_1,'date_time'] <= datetime.timedelta(minutes=i_time_step)],:]


        if len(table_data_3.index) > 1 :

            table_data_1.iloc[aux_counter_1,3] = datetime.timedelta.total_seconds(table_data_3.iloc[-1,1] - table_data_3.iloc[0,1])/60
            if table_data_1.iloc[aux_counter_1,3] >= i_time_step:
                table_data_1.iloc[aux_counter_1,4] = table_data_3.loc[:,'value'].sum() - table_data_3.iloc[-1,0]
            else:
                table_data_1.iloc[aux_counter_1,4] = table_data_3.loc[:,'value'].sum()
            table_data_1.iloc[aux_counter_1,5] = table_data_3.iloc[-1,1]

        elif len(table_data_3.index) == 1 :
            table_data_1.iloc[aux_counter_1,3] = 0
            table_data_1.iloc[aux_counter_1,4] = table_data_3.loc[:,'value'].sum()
            table_data_1.iloc[aux_counter_1,5] = table_data_3.iloc[-1,1]
        else:
                print(table_data_3)


    table_data_1["year_stamp"] = table_data_1["datetime_stamp"].dt.year

    table_data_2 = table_data_2

    return table_data_2

Upvotes: 2

Views: 154

Answers (1)

cco
cco

Reputation: 6301

I think the issue is hat numpy is creating a large number of threads, assuming it is the only process running on the machine.

See Limit number of threads in numpy for how to limit the number of threads used.

Upvotes: 1

Related Questions