Remove row or column from 2D list if all values (in that row or column) are None

Question

I have a grid (6 rows, 5 columns):

grid = [
        [None, None, None, None, None],
        [None, None, None, None, None],
        [None, None, None, None, None],
        [None, None, None, None, None],
        [None, None, None, None, None],
        [None, None, None, None, None],
        ]

I augment the grid and it might turn into something like:

grid = [
        [{"some" : "thing"}, None, None, None, None],
        [None, None, None, None, None],
        [None, None, None, None, None],
        [None, None, None, {"something" : "else"}, None],
        [None, {"another" : "thing"}, None, None, None],
        [None, None, None, None, None],
        ]

I want to remove entire rows and columns that have all Nones in them. So in the previous code, grid would be transformed into:

grid = [
        [{"some" : "thing"}, None, None],
        [None, None, {"something" : "else"}],
        [None, {"another" : "thing"}, None],
        ]

I removed row 1, 2, 5 (zero indexed) and column 2 and 4.

The way I am deleting the rows now:

for row in range(6):
    if grid[row] == [None, None, None, None, None]:
        del grid[row] 

I don't have a decent way of deleting None columns yet. Is there a "pythonic" way of doing this?

Answer 1

Here is quick attempt

It will work for any size matrix and rows can be different sizes too , and may be fast :)

from collections import defaultdict
grid = [
        [{"some" : "thing"}, None, None, None, None],
        [None, None, None, None, None],
        [None, None, None, None, None],
        [None, None, None, {"something" : "else"}, None],
        [None, {"another" : "thing"}, None, None, None],
        [None, None, None, None, None],
        ]

# go thru the grid remove, rows which have all None
# doing that count None in each columns, remove such columns later
newGrid = []
colSize = len(grid)
colCount = defaultdict(int)
for row in grid:
    allNone = True
    for c, cell in enumerate(row):
        if cell is None:
            colCount[c] += 1
        else:
            allNone = False

    if not allNone: # only add rows which are not all none
        newGrid.append(row)

# get cols which need to be removed
removeCols = [col for col, count in colCount.iteritems() if count == colSize]
removeCols.sort(reverse=True) 

# now go thru each column and remove all None Columns
for row in newGrid:
    for col in removeCols:
        row.pop(col)

grid = newGrid
import pprint
pprint.pprint(grid)

output:

[[{'some': 'thing'}, None, None],
 [None, None, {'something': 'else'}],
 [None, {'another': 'thing'}, None]]

Answer 2

You can also use the built-in function any(), which checks if any of the elements of an iterable has a not None value. It's faster than comparing and you don't need to know the size of the iterable.

>>> def remove_rows( matrix ):
...    '''Returns a matrix without empty rows'''
...    ret_matrix = []
...    for row in matrix:
...        #Check if the row has any value or all are None
...        if any(row):
...            ret_matrix.append(row)
...
...    return ret_matrix
#You can do it also with a list comprehension, which will be even faster
>>> def remove_rows(matrix):
...     '''Returns a matrix without empty rows'''
...     ret_matrix = [ row for row in matrix if  any(row) ]
...     return ret_matrix

>>> grid = [
...         [{"some" : "thing"}, None, None, None, None],
...         [None, None, None, None, None],
...         [None, None, None, None, None],
...         [None, None, None, {"something" : "else"}, None],
...         [None, {"another" : "thing"}, None, None, None],
...         [None, None, None, None, None],
...         ]


>>> grid = remove_rows( grid )
>>> grid
[
 [{'some': 'thing'}, None, None, None, None], 
 [None, None, None, {'something': 'else'}, None], 
 [None, {'another': 'thing'}, None, None, None]
]

#transpose grid using zip (using asterisk)
>>> grid = zip(*grid)
#Note that zip returns tuples
>>> grid
[
  ({'some': 'thing'}, None, None), 
  (None, None, {'another': 'thing'}),
  (None, None, None), 
  (None, {'something': 'else'}, None), 
  (None, None, None)
]

>>> grid = remove_rows(grid)
>>> grid
[
  ({'some': 'thing'}, None, None), 
  (None, None, {'another': 'thing'}),
  (None, {'something': 'else'}, None)
]
>>> #Transpose again to get the first matrix, without empty rows or columns
>>> final_grid = zip(*grid)
>>> final_grid
[
 ({'some': 'thing'}, None, None),
 (None, None, {'something': 'else'}), 
 (None, {'another': 'thing'}, None)
]

Answer 3

It's not the fastest way but I think it's quite easy to understand:

def transpose(grid):
    return zip(*grid)

def removeBlankRows(grid):
    return [list(row) for row in grid if any(row)]

print removeBlankRows(transpose(removeBlankRows(transpose(grid))))

Output:

[[{'some': 'thing'}, None, None],
 [None, None, {'something': 'else'}],
 [None, {'another': 'thing'}, None]]

How it works: I use zip to write a function that transposes the rows and columns. A second function removeBlankRows removes rows where all elements are None (or anything that evaluates to false in a boolean context). Then to perform the entire operation I transpose the grid, remove blank rows (which are the columns in the original data), transpose again, then remove blank rows.

If it's important to only strip None and not other things that evaluate to false, change the removeBlankRows function to:

def removeBlankRows(grid):
    return [list(row) for row in grid if any(x is not None for x in row)]

Answer 4

If only you had a transpose function, you could do: transpose(removeRows(transpose(removeRows(mat))))

Actually ... using a bitmask is a better idea.

Let me think about that ...

First compute gridmask:

grid_mask = [
10000,
00000,
00000,
00010,
00000
]

Now remove zeroes:

grid_mask = [
10000,
00010,
]

Now and all values bitwise:

grid_mask = 10010

Now remove all but 1st and 4th columns.

Answer 5

Use zip() to transpose the ragged array, run the clearing routine again, then zip() it again.

Answer 6

grid = ...

# remove empty rows
grid = [x for x in grid if any(x)]
# if any value you put in won't evaluate to False
# e.g. an empty string or empty list wouldn't work here
# in that case, use:
grid = [x for x in grid if any(n is not None for n in x)]

# remove empty columns
if not grid:
  raise ValueError("empty grid")
  # or whatever, as next line assumes grid[0] exists
empties = range(len(grid[0])) # assume all empty at first
for r in grid:
  empties = [c for c in empties if r[c] is None] # strip out non-empty
if empties:
  empties.reverse() # apply in reversed order
  for r in grid:
    for c in empties:
      r.pop(c)