CODEWITHSUNDEEP
pythonpython-3.xpandaswarnings
jakeanq
jakeanq

Reputation: 127

Filtering third-party Python warnings with newlines

I am trying to suppress a warning from a third-party module (in this case, PyTables via Pandas) using an environment variable. The warning starts with a newline.

Example of warning:

python -c 'import pandas as pd; pd.DataFrame([None]).to_hdf("test.h5", "/data")'
/usr/lib/python3.9/site-packages/pandas/core/generic.py:2606: PerformanceWarning: 
your performance may suffer as PyTables will pickle object types that it cannot
map directly to c-types [inferred_type->mixed,key->block0_values] [items->Int64Index([0], dtype='int64')]

  pytables.to_hdf(

According to the Python documentation I can specify a warning to ignore by setting PYTHONWARNINGS in the format action:message:category:module:line

Using category

I tried PYTHONWARNINGS=ignore::PerformanceWarning, however this gives me the message Invalid -W option ignored: unknown warning category: 'PerformanceWarning'.

PYTHONWARNINGS=ignore::pytables.PerformanceWarning gives me Invalid -W option ignored: invalid module name: 'pytables'.

Using message

I initially tried PYTHONWARNINGS=ignore:your performance may suffer which does nothing, likely due to the warning message starting with a newline.

Further exploring the newline case, I tried to insert a newline, however it is either escaped or removed:

PYTHONWARNINGS="ignore:\nyour performance" python -c "import warnings as w; [print(f) for f in w.filters]"
('ignore', re.compile('\\\\nyour\\ performance', re.IGNORECASE), <class 'Warning'>, None, 0)
// ...

PYTHONWARNINGS="$(echo -n "ignore:\nyour performance")" python -c "import warnings as w; [print(f) for f in w.filters]"
('ignore', re.compile('your\\ performance', re.IGNORECASE), <class 'Warning'>, None, 0)
// ...

PYTHONWARNINGS="ignore:.your performance" python -c "import warnings as w; [print(f) for f in w.filters]"
('ignore', re.compile('\\.your\\ performance', re.IGNORECASE), <class 'Warning'>, None, 0)
// ...

How can I disable this warning?

(I am aware that I can do it using something like python -c 'import pandas as pd; import warnings as w; w.filterwarnings("ignore", "\nyour performance"); pd.DataFrame([None]).to_hdf("test.h5", "/data")', however it is quite inconvenient for my use case. I could also solve the underlying issue, but that is not a small task)

Upvotes: 1

Views: 876

Answers (1)

Jason
Jason

Reputation: 4546

Your problem occurs because PerformanceWarning isn't a built-in warning category (list here).

Based on your use case you can use an approach below, but be careful as some heavy-handed approaches may also suppress useful warnings.

If you want to ignore all warnings when running a script from the command line you can use:

python -W ignore myscript.py

Your example:

python -W ignore -c 'import pandas as pd; pd.DataFrame([None]).to_hdf("test.h5", "/data")'

Inside of a script you can use:

import pandas as pd
import warnings

# Create your own warning filter
warnings.simplefilter(action='ignore', category=pd.errors.PerformanceWarning)

Your example:

python -c 'import warnings; import pandas as pd; warnings.simplefilter(action='ignore', category=pd.errors.PerformanceWarning); pd.DataFrame([None]).to_hdf("test.h5", "/data")'

I would recommend using some variation of the second approach where you explicitly ignore the specific warning(s) you don't want to see.

Upvotes: 1

Related Questions