CODEWITHSUNDEEP
ctype-conversionbit-manipulation
christofferp
christofferp

Reputation: 37

bitmask + typecasting does not return as expected

I have an situation where bitmasking and/or typecasting does not work as expected. I am working with an overwhelmingly big distributed system where some parts are written i C.

The few lines of code that confuses me are these:

uint16_t
mask(uint32_t nr)
{
  return (uint16_t)(nr & 0x1ff);
}

...

  int nr = 65536;
  int index;
  index = mask(nr);

...

In this case the return of mask is 65536 and not 0 as expected. There are some inconsistent type conversions in this example, but I cannot understand how the result could be wrong. We are basically masking out the lower 9 bits of an 32 bit integer and converting the result to an 16 bit integer. My best guess is that there are some compiler optimizations that changes the first 16 bits, but does nothing about the higher 16 bits since we only want to return 16 bits. Since the 'index' in this case is an 32 bit integer this also includes the 17:th bit of 'nr', which was not modified in the bitmask, thus the 17:th bit is still set to 1 and assigning 65536 to index.

Is there anyone that have an explanation of this behavior?

Edit: I forgot some important parameters. The prototype of the mask function is properly included. The system is built for Linux x86 with gcc 4.3.2.

Edit2: Assembly code:

0805b640 <mask>:
 805b640:       55                      push   %ebp
 805b641:       89 e5                   mov    %esp,%ebp
 805b643:       83 ec 18                sub    $0x18,%esp
 805b646:       65 a1 14 00 00 00       mov    %gs:0x14,%eax
 805b64c:       89 45 fc                mov    %eax,0xfffffffc(%ebp)
 805b64f:       31 c0                   xor    %eax,%eax
 805b651:       8b 45 08                mov    0x8(%ebp),%eax
 805b654:       66 25 ff 01             and    $0x1ff,%ax
 805b658:       8b 55 fc                mov    0xfffffffc(%ebp),%edx
 805b65b:       65 33 15 14 00 00 00    xor    %gs:0x14,%edx
 805b662:       75 02                   jne    805b666 <maks+0x26>
 805b664:       c9                      leave  
 805b665:       c3                      ret    
 805b666:       e8 a5 ac 04 00          call   80a6310 <__stack_chk_fail_local>
 805b66b:       90                      nop    
 805b66c:       8d 74 26 00             lea    0x0(%esi),%esi

Upvotes: 1

Views: 354

Answers (2)

Daniel Fischer
Daniel Fischer

Reputation: 183968

Here's what my gcc (4.6.2) produces from mask (with optimisations; without, there's some fluff around it):

.LFB0:
    .cfi_startproc
    movl    4(%esp), %eax
    andw    $511, %ax
    ret
    .cfi_endproc

So it indeed only modifies the lower order 16 bits, and it is the responsibility of the caller to actually zero out the high-order 16 bits per

movzwl  %ax, %eax

which it does if the correct prototype is known at the call site. I very strongly suspect that

The prototype of the mask function is properly included.

is not actually true, check and re-check. If it's true indeed, you have found a major bug (so major that we'd probably have heard about it).

uint16_t and uint32_t are provided by including linux/types.h

Hmm, not here. They're provided by stdint.h. You should #include <stdint.h> to get them anyway, even if they're also provided by linux/types.h on your system, since that's the portable solution.

Upvotes: 2

hr_117
hr_117

Reputation: 9627

The only possible explanation I see here,is that not the expected mask function is called. Please try your example code in one file only, and/or rename the mask function. The bit and (nr & 0x1ff) for your value (0x10000) should always lead to zero.

Upvotes: 0

Related Questions