CODEWITHSUNDEEP
luabit-manipulation
hugomg
hugomg

Reputation: 69934

How can I convert a pair of bytes into a signed 16-bit integer using Lua?

I am writing a program that needs to serialize small (up to 16 bit) signed integers to a binary file. As part of that, I need to write a function that can split an integer into two bytes and another function that can convert this pair of bytes back to the original integer.

The first idea that came to my mind was to solve this problem similarly to how I would solve it C:

function dump_i16(n)
  assert (-0x8000 <= n and n < 0x8000) -- 16 bit
  local b1 = (n >> 8) & 0xff
  local b2 = (n >> 0) & 0xff
  return b1, b2
end

function read_i16(b1, b2)
  assert (0 <= b1 and b1 <= 0xff) -- 8 bit
  assert (0 <= b2 and b2 <= 0xff) -- 8 bit
  return (b1 << 8) | (b2 << 0)
end

However, these functions break on negative numbers because Lua integers actually have 64 bits and I am only preserving the lower 16 bits:

-- Positive numbers are OK
print( read_i16(dump_i16(17)) ) -- 17
print( read_i16(dump_i16(42)) ) -- 42

-- Negative numbers don't round trip.
print( read_i16(dump_i16(-1)) )  -- 65535 = 2^16 - 1
print( read_i16(dump_i16(-20)) ) -- 65516 = 2^16 - 20

What would be the cleanest way to modify my read_i16 function so it works correctly for negative numbers?

I would prefer to do this using pure Lua 5.3 if possible, without going down to writing C code.

Upvotes: 3

Views: 8020

Answers (3)

hugomg
hugomg

Reputation: 69934

One way to fix read_i16 is to use the sign extension algorithm that is used internally by string.pack:

function read_i16(b1, b2)
  assert (0 <= b1 and b1 <= 0xff)
  assert (0 <= b2 and b2 <= 0xff)
  local mask = (1 << 15)
  local res  = (b1 << 8) | (b2 << 0)
  return (res ~ mask) - mask
end

Upvotes: 0

Egor Skriptunoff
Egor Skriptunoff

Reputation: 23727

Lua 5.3 has special conversion functions.

To convert an integer -32768..32767 to string of length 2 in big-endian order: 

local a_string = string.pack(">i2", your_integer)

To convert it back (first two bytes of "a_string" are being converted):

local an_integer = string.unpack(">i2", a_string)

Upvotes: 4

Moop
Moop

Reputation: 3601

You need a logical right shift for dealing with the sign. See Difference between >>> and >>. Lua doesn't have it, so you have do your own. Since Lua integers are 64 by default, you mask off 64 minus the number of shifts you do.

function dump_i16(n)
   assert (-0x8000 <= n and n < 0x8000)
   local b1 = (n >> 8) & ~(-1<<(64-8))
   local b2 = (n >> 0) & 0xff
   return b1, b2
end

Upvotes: 1

Related Questions