CODEWITHSUNDEEP
pythonpython-3.xsuppress-warnings
mmann1123
mmann1123

Reputation: 5295

Show only first of repeated warnings python

I am doing a distributed process in dask, although I appreciate a good warning here and there, it has tends to send a warning each time a worker is close to it memory limit. Which happens quite a bit.

Just wondering if there is a clever way to not show the first warning and suppress all duplicate warnings. Given that the process is distributed, I am assuming no, but worth asking.

from tsfresh.examples.robot_execution_failures import \
download_robot_execution_failures, \
load_robot_execution_failures
from tsfresh.feature_extraction import extract_features
from tsfresh.utilities.distribution import LocalDaskDistributor
import copy
import pandas as pd


download_robot_execution_failures()
df, y = load_robot_execution_failures()

df_comb = copy.deepcopy(df)
for x in range(100):
    df_iter = df
    df_iter['id'] = df_iter['id'] + df_comb['id'].max()
    df_comb = pd.concat([df_comb, df_iter], axis = 0)
    print(len(df_comb))

df_comb.reset_index(inplace = True, drop = True)


Distributor = LocalDaskDistributor(n_workers=3)

X = extract_features(timeseries_container=df_comb,
                     column_id='id', column_sort='time',
                     distributor=Distributor)

distributed.worker - WARNING - Memory use is high but worker has no data to store to disk.  Perhaps some other process is leaking memory?  Process memory: 94.87 GB -- Worker memory limit: 134.76 GB
distributed.worker - WARNING - Memory use is high but worker has no data to store to disk.  Perhaps some other process is leaking memory?  Process memory: 94.88 GB -- Worker memory limit: 134.76 GB
distributed.worker - WARNING - Memory use is high but worker has no data to store to disk.  Perhaps some other process is leaking memory?  Process memory: 95.02 GB -- Worker memory limit: 134.76 GB

Upvotes: 0

Views: 750

Answers (1)

Bastien Harkins
Bastien Harkins

Reputation: 305

Is this what you're looking for?

from tsfresh.utilities.distribution import initialize_warnings_in_workers

initialize_warnings_in_workers(False)

Here is what this convenient function does:

def initialize_warnings_in_workers(show_warnings):
    """
    Small helper function to initialize warnings module in multiprocessing workers.

    On Windows, Python spawns fresh processes which do not inherit from warnings
    state, so warnings must be enabled/disabled before running computations.

    :param show_warnings: whether to show warnings or not.
    :type show_warnings: bool
    """
    warnings.catch_warnings()
    if not show_warnings:
        warnings.simplefilter("ignore")
    else:
        warnings.simplefilter("default")

Upvotes: 1

Related Questions