CODEWITHSUNDEEP
c++functionassemblyx86stack
cloudrain21
cloudrain21

Reputation: 659

Program was killed after function returning

My program was killed when the function "dbmSetIndex" was returned. You can see the killed position below. (=> part) Could you explain what happened? (x86 intel)

   0x0000000000420723 <+83>:    mov    rdi,r14
   0x0000000000420726 <+86>:    call   0x405260 <dbmSetIndex@plt>
   0x000000000042072b <+91>:    test   eax,eax
   0x000000000042072d <+93>:    mov    ebp,eax
=> 0x000000000042072f <+95>:    mov    DWORD PTR [r12],eax
   0x0000000000420733 <+99>:    jne    0x4207d0 <FnDBBase::SelectSecurity(s_oms_security*, char*)+256>
   0x0000000000420739 <+105>:   lea    rsi,[rip+0x4197d]        # 0x4620bd

Here is the dbmSetIndex code. I can't find what part of this code caused this problem.

int dbmSetIndex ( dbmHandle* aHandle, const char* aTable, const char* aIndexName )
{
    dbmInternalHandle* pHandle = NULL;

    _TRY
    {   
        pHandle = (dbmInternalHandle*)aHandle->mHandle;

        // Water Mark가 다르면 걍 리턴해라.
        if ( unlikely ( aHandle->mMark != DBM_HANDLE_MARK ) ) 
        {   
            DBM_ERR ( "invalide magic number (%ld)", aHandle->mMark );
            _THROW( ERR_DBM_INVALID_HANDLE );
        }   

        if( pHandle->mRemote != NULL )
        {   
            if( pHandle->mRemote->mSockFd > 0 ) 
            {
                _CALL( dbmRemoteSetIndex( aHandle, aTable, aIndexName ) );
                _RETURN;
            }
        }   

        /****************************************
         * DataObject 맵핑.
        ****************************************/
        memset_s( &pHandle->mData, 0x00, sizeof( pHandle->mData ) );
        memcpy_s ( pHandle->mData.mTableName, aTable, strlen_s ( aTable ) + 1 );
        pHandle->mData.mTransType = DBM_SET_INDEX;
        pHandle->mData.mUserData  = (char*)aIndexName;

        /****************************************
         * mAct 호출.
        ****************************************/
        _CALL( pHandle->mTrans->mAct ( pHandle ) );
    }   
    _CATCH
    {   
        _CATCH_ERR;
    }   
    _FINALLY
    _END
}

Upvotes: 0

Views: 59

Answers (1)

Aconcagua
Aconcagua

Reputation: 25526

You provided a little bit of dissassembly which shows that the call to the function you provided returned already. The crash did not occur within the function, but afterwards:

Call of your function:

0x0000000000420726 <+86>:    call   0x405260 <dbmSetIndex@plt>

Here you returned already:

0x000000000042072b <+91>:    test   eax,eax

The critical line is a memory access to the address stored in register r12 (write):

0x000000000042072f <+95>:    mov    DWORD PTR [r12],eax

Let your debugger show the registers and have a look at the content of r12. It is very likely that it is 0x0000000000000000 or a small value, thus a null pointer (or null reference), but it could contain, too, an invalid address (uninitialised pointer!).

There is little more left to tell from the information you provide, though. You will have to look at the code location where the function is called -- it must be within function FnDBBase::SelectSecurity(s_oms_security*, char*) because you jump (jne) to an offset within this function ([...] + 256). There should be an if involved (test + jne instructions) and probably some pointer assignment. Possibly something like this: 

SomeClass* s = [...];
s->someMember = dbmSetIndex([...]); // (*)
if(*s->someMember)

(*): Failure location, error happened during assignment after the function completed already. Don't count 100% on finding such an assignment, it could be, too, a call to an inlined setter function.

And here we see, too, why r12 is not necessarily 0: it would contain the precalculated offset of someMember within SomeClass, i. e. &(s->someMember) which could well be, if s is 0, e. g. 16, 28, ... Precisely, the value contained then is equal to offsetof(SomeClass, somemember).

Upvotes: 1

Related Questions