CODEWITHSUNDEEP
pythonperformancelookup
Alexii
Alexii

Reputation: 58

Efficient lookup table for bytes

I need to unwrap raw bytes to bits. Now I have a raw data and a lookup table. What is the most efficient way to iterate over input data to generate output? Or maybe there is another way to do this?

#Look up table looks something like this.
lookup = {
  0: b'\x00\x00\x00\x00\x00\x00\x00\x00',
  1: b'\x00\x00\x00\x00\x00\x00\x00\x01',
  2: b'\x00\x00\x00\x00\x00\x00\x01\x00',
  ...
  255: b'\x01\x01\x01\x01\x01\x01\x01\x01',
}

def remap(data):
  out = [lookup(byte) for byte in data]
  row = b''.join(out)

Here are functions that take most time:

ncalls  tottime  percall  cumtime  percall filename:lineno(function)
44000    2.843    0.000    2.843    0.000 main.py:59(<listcomp>)
44007    0.593    0.000    0.593    0.000 {method 'join' of 'bytes' objects}

Upvotes: 1

Views: 420

Answers (1)

Pik-Mai Hui
Pik-Mai Hui

Reputation: 339

Turns out my guesses are quite wrong. But the comments have interesting readings for why they are so.

Here I am giving two trivial improvement that hopefully may improve the runtime performance a bit.

First, your lookup table has natural numbers as keys. It is a list.

lookup = [
  b'\x00\x00\x00\x00\x00\x00\x00\x00',
  b'\x00\x00\x00\x00\x00\x00\x00\x01',
  b'\x00\x00\x00\x00\x00\x00\x01\x00',
  ...
  b'\x01\x01\x01\x01\x01\x01\x01\x01',
]

Second, instead of constructing the list and then input it to join, use generator.

def remap(data):
    return b''.join(lookup[byte] for byte in data)

But you may want to test ideas in this question as well:

Converting integer to binary in python

Maybe this fits your need as well, but it gives list instead of bstring.

https://docs.scipy.org/doc/numpy-1.13.0/reference/generated/numpy.unpackbits.html#numpy.unpackbits

Upvotes: 2

Related Questions