CODEWITHSUNDEEP
pythonarraysnumpymask
Josiah
Josiah

Reputation: 329

Creating a masked array in Python with multiple given values

I am graphing several columns of a large array of data (through numpy.genfromtxt) against an equally sized time column. Missing data is often referred to as nan, -999, -9999, etc. However I can't figure out how to remove multiple values from the array. This is what I currently have:

for cur_col in range(start_col, total_col):
    # Generate what is to be graphed by removing nan values
    data_mask = (file_data[:, cur_col] != nan_values)
    y_data = file_data[:, cur_col][data_mask]
    x_data = file_data[:, time_col][data_mask]

After which point I use matplotlib to create the appropriate figures for each column. This works fine if the nan_values is a single integer, but I am looking to use a list.

EDIT: Here is a working example.

import numpy as np

file_data = np.arange(12.0).reshape((4,3))
file_data[1,1] = np.nan
file_data[2,2] = -999
nan_values = -999

for cur_col in range(1,3):
    # Generate what is to be graphed by removing nan values
    data_mask = (file_data[:, cur_col] != nan_values)
    y_data = file_data[:, cur_col][data_mask]
    x_data = file_data[:, 0][data_mask]
    print 'y: ' + str(y_data)
    print 'x: ' + str(x_data)
print file_data

>>> y: [  1.  nan   7.  10.]
    x: [ 0.  3.  6.  9.]
    y: [  2.   5.  11.]
    x: [ 0.  3.  9.]
    [[   0.    1.    2.]
    [   3.   nan    5.]
    [   6.    7. -999.]
    [   9.   10.   11.]]

This will not work if nan_values = ['nan', -999] which is what I am looking to accomplish.

Upvotes: 5

Views: 8867

Answers (2)

user545424
user545424

Reputation: 16224

I would suggest using masked arrays like so:

>>> a = np.arange(12.0).reshape((4,3))
>>> a[1,1] = np.nan
>>> a[2,2] = -999
>>> a
array([[   0.,    1.,    2.],
       [   3.,   nan,    5.],
       [   6.,    7., -999.],
       [   9.,   10.,   11.]])
>>> m = np.ma.array(a,mask=(~np.isfinite(a) | (a == -999)))
>>> m
masked_array(data =
 [[0.0 1.0 2.0]
 [3.0 -- 5.0]
 [6.0 7.0 --]
 [9.0 10.0 11.0]],
             mask =
 [[False False False]
 [False  True False]
 [False False  True]
 [False False False]],
       fill_value = 1e+20)

Upvotes: 7

GL770
GL770

Reputation: 3210

I would try something like (pseudo-code):

nan_values = [...]

for cur_col in range(start_col, total_col):
    # Generate what is to be graphed by removing nan values
    y_data = [file_data[i,cur_col] for i in range(len(file_data)) if not(file_data[i,cur_col] in nan_values)]
    x_data = [file_data[i,time_col] for i in range(len(file_data)) if not(file_data[i,cur_col] in nan_values)]

Upvotes: 2

Related Questions