CODEWITHSUNDEEP
c++structdeserializationendianness
Abhishek Bansal
Abhishek Bansal

Reputation: 31

Change endianness of entire struct in C++

I am writing a parser in C++ to parse a well defined binary file. I have declared all the required structs. And since only particular fields are of interest to me, so in my structs I have skipped non-required fields by creating char array of size equal to skipped bytes. So I am just reading the file in char array and casting the char pointer to my struct pointer. Now problem is that all data fields in that binary are in big endian order, so after typecasting I need to change the endianness of all the struct fields. One way is to do it manually for each and every field. But there are various structs with many fields, so it'll be very cumbersome to do it manually. So what's the best way to achieve this. And since I'll be parsing very huge such files (say in TB's), so I require a fast way to do this.

EDIT : I have use attribute(packed) so no need to worry about padding.

Upvotes: 0

Views: 1562

Answers (3)

Soonts
Soonts

Reputation: 21966

One way to do that is combine C preprocessor with C++ operators. Write a couple of C++ classes like this one:

#include "immintrin.h"

class FlippedInt32
{
    int value;
public:
    inline operator int() const
    {
        return _bswap( value );
    }
};

class FlippedInt64
{
    __int64 value;
public:
    inline operator __int64() const
    {
        return _bswap64( value );
    }
};

Then, 

#define int FlippedInt32

before including the header that define these structures. #undef immediately after the #include.

This will replace all int fields in the structures with FlippedInt32, which has the same size but returns flipped bytes.

If it’s your own structures which you can modify you don’t need the preprocessor part. Just replace the integers with the byte-flipping classes.

Upvotes: 1

Jeremy Friesner
Jeremy Friesner

Reputation: 73334

If you can come up with a list of offsets (in-bytes, relative to the top of the file) of the fields that need endian-conversion, as well as the size of those fields, then you could do all of the endian-conversion with a single for-loop, directly on the char array. E.g. something like this (pseudocode):

struct EndianRecord {
   size_t offsetFromTop;
   size_t fieldSizeInByes;
};

std::vector<EndianRecord> todoList;
// [populate the todo list here...]

char * rawData = [pointer to the raw data]
for (size_t i=0; i<todoList.size(); i++)
{
   const EndianRecord & er = todoList[i];
   ByteSwap(&rawData[er.offsetFromTop], er.fieldSizeBytes);
}

struct MyPackedStruct * data = (struct MyPackedStruct *) rawData;
// Now you can just read the member variables
// as usual because you know they are already
// in the correct endian-format.

... of course the difficult part is coming up with the correct todoList, but since the file format is well-defined, it should be possible to generate it algorithmically (or better yet, create it as a generator with e.g. a GetNextEndianRecord() method that you can call, so that you don't have to store a very large vector in memory)

Upvotes: 0

Useless
Useless

Reputation: 67822

If you can do misaligned accesses with no penalty, and you don't mind compiler- or platform-specific tricks to control padding, this can work. (I assume you are OK with this since you mention __attribute__((packed))).

In this case the nicest approach is to write value wrappers for your raw data types, and use those instead of the raw type when declaring your struct in the first place. Remember the value wrapper must be trivial/POD-like for this to work. If you have a POSIX platform you can use ntohs/ntohl for the endian conversion, it's likely to be better optimized that whatever you write yourself.

If misaligned accesses are illegal or slow on your platform, you need to deserialize instead. Since we don't have reflection yet, you can do this with the same value wrappers (plus an Ignore<N> placeholder that skips N bytes for fields you're not interested), and declare them in a tuple instead of a struct - you can iterate over the members in a tuple and tell each to deserialize itself from the message.

Upvotes: 1

Related Questions