numpy parse int into bit groupings

Question

I have a np.array of np.uint8

a = np.array([randint(1,255) for _ in range(100)],dtype=np.uint8)

and I want to split this into low and high nibbles

I could get the low nibble

low = np.bitwise_and(a,0xF)

and I could get the high nibble with

high = np.bitwise_and(np.right_shift(a,4),0xF)

is there some way to do something like

>>> numpy.keep_bits(a,[(0,3),(4,7)])
numpy.array([
  [low1,high1],
  [low2,high2],
  ...
  [lowN,highN]
  ])

Im not even sure what this would be called ... but I thought maybe some numpy guru would know a cool way to do this (in reality i am looking to do this with uint32's and much more varied nibbles

basically something like struct.unpack but for vectorized numpy operations

EDIT: I went with a modified version of the accepted answer below

here is my final code for anyone interested

def bitmask(start,end):
    """
    >>> bitmask(0,2) == 0b111
    >>> bitmask(3,5) == 0b111000

    :param start: start bit 
    :param end:  end bit (unlike range, end bit is inclusive)
    :return: integer bitmask for the specified bit pattern
    """
    return (2**(end+1-start)-1)<<start

def mask_and_shift(a,mask_a,shift_a):
    """

    :param a: np.array 
    :param mask_a: array of masks to apply (must be same size as shift_a)
    :param shift_a: array of shifts to apply (must be same size as mask_a)
    :return: reshaped a, that has masks and shifts applied
    """
    masked_a = numpy.bitwise_and(a.reshape(-1,1), mask_a)
    return numpy.right_shift(masked_a,shift_a)

def bit_partition(rawValues,bit_groups):
    """
    >>> a = numpy.array([1,15,16,17,125,126,127,128,129,254,255])
    >>> bit_partition(a,[(0,2),(3,7)])
    >>> bit_partition(a,[(0,2),(3,5),(6,7)])

    :param rawValues: np.array of raw values
    :param bit_groups: list of start_bit,end_bit values for where to bit twiddle
    :return: np.array len(rawValues)xlen(bit_groups)
    """
    masks,shifts = zip(*[(bitmask(s,e),s) for s,e in bit_groups])
    return mask_and_shift(rawValues,masks,shifts)

Answer 1

A one-liner, using broadcasting, for the four bit lower and upper nibbles:

In [38]: a
Out[38]: array([  1,  15,  16,  17, 127, 128, 255], dtype=uint8)

In [39]: (a.reshape(-1,1) & np.array([0xF, 0xF0], dtype=np.uint8)) >> np.array([0, 4], dtype=np.uint8)
Out[39]: 
array([[ 1,  0],
       [15,  0],
       [ 0,  1],
       [ 1,  1],
       [15,  7],
       [ 0,  8],
       [15, 15]], dtype=uint8)

To generalize this, replace the hardcoded values [0xF, 0xF0] and [0, 4] with the appropriate bit masks and shifts. For example, to split the values into three groups, containing the highest two bits, followed by the remaining two groups of three bits, you can do this:

In [41]: masks = np.array([0b11000000, 0b00111000, 0b00000111], dtype=np.uint8)

In [42]: shifts = np.array([6, 3, 0], dtype=np.uint8)

In [43]: a
Out[43]: array([  1,  15,  16,  17, 127, 128, 255], dtype=uint8)

In [44]: (a.reshape(-1,1) & np.array(masks, dtype=np.uint8)) >> np.array(shifts, dtype=np.uint8)
Out[44]: 
array([[0, 0, 1],
       [0, 1, 7],
       [0, 2, 0],
       [0, 2, 1],
       [1, 7, 7],
       [2, 0, 0],
       [3, 7, 7]], dtype=uint8)

Answer 2

So, I won't comment on the specific logical operators you want to implement, since bit-hacking isn't quite a specialty of mine, but I can tell you where you should look in numpy to implement this kind of custom operator.

If you look through the numpy source you'll notice that nearly all of the bit-maniuplation techniques in numpy are just instances of _MaskedBinaryOperation, for example, the definition of bitwise_and is simply:

bitwise_and = _MaskedBinaryOperation(umath.bitwise_and)

The magic here comes in the form of the umath module, which calls down, typically to the low level libraries that numpy is built on. If you really want to, you could add your operator there, but I don't think it's worth mucking around at that level.

That said, this isn't the only way to incorporate these functions into numpy. In fact, the umath module has a really handy function called frompyfunc that will let you turn an arbitrary python function into one of these handy umath operators. Documentation can be found here. An example of creating such a function is below:

>>> oct_array = np.frompyfunc(oct, 1, 1)
>>> oct_array(np.array((10, 30, 100)))
array([012, 036, 0144], dtype=object)
>>> np.array((oct(10), oct(30), oct(100))) # for comparison
array(['012', '036', '0144'],
      dtype='|S4')

If you decide on the specifics of the bitwise operator you want to implement, using this interface would be the best way to implement it.

This doesn't answer 100% of your question, but I assumed your question was much more about implementing some custom bitwise operator in proper numpy form rather than digging into the bitwise operator itself. Let me know if that's inaccurate and I can put together an example using the bitwise operator you alluded to above.