Load file contents into a C++ struct

Question

I have an ASCII file which looks like this (the file is large, so pasting only partial contents):

ffffffffffffffff0064000a000a000c
02000000000000000b0000111f0a0503
00000000000002000000000000000000
00000000000000000000000000000000
02000000000000000000020400000000
00000000000000000000000000000000
00000000000000000000000000000000
ffffffffffffffffffffffff00000000
ffffffffffffffffffffffffffffffff
ffffffffffffffffffffffffffffffff
ffffffffffffffffffffffffffffffff
ffffffffffffffffffffffffffffffff

I have a struct that looks like this:

typedef struct frame {

  uint16_t kps;
  uint16_t num_h_region;
  uint16_t num_v_region;
  uint16_t num_r;
  uint8_t reserved1[7];
  uint8_t kp_per_region[100];
  uint8_t reserved[397];

  kp_info_t kp_info[2000];
  uint32_t desc[2000];

} frame_t;

Where:

typedef struct kp_info {
  uint32_t subpixel_idx_x;
  uint32_t subpixel_idx_y;
  uint32_t orientation;
  uint32_t laplacian;
  uint8_t scale;
  uint8_t minima; // 1 bit
  uint8_t dmy1[14];
  uint32_t subpixel_match0_x;
  uint32_t subpixel_match0_y;
  uint32_t subpixel_match1_x;
  uint32_t subpixel_match1_y;
  uint32_t distance_match0;
  uint32_t distance_match1;
  uint8_t ambiguous; // 1 bit
  uint8_t dmy2[7];
} kp_info_t;

The input file is expected to be assigned to the frame_t struct members like this:

frame_t fr;
fr.kps = 0x000c;
fr.num_h = 0x000a;
fr.num_v = 0x000a;
fr.num_r = 0x0064;
fr.reserved1[0] = 0xff;
fr.reserved1[1] = 0xff;
fr.reserved1[2] = 0xff;
fr.reserved1[3] = 0xff;
fr.reserved1[4] = 0xff;
fr.reserved1[5] = 0xff;
fr.reserved1[6] = 0xff;
fr.reserved1[7] = 0xff;
fr.kps_per_region[0] = 0x03;
fr.kps_per_region[1] = 0x05;
fr.kps_per_region[2] = 0x0a;
fr.kps_per_region[3] = 0x1f;
fr.kps_per_region[4] = 0x11;
fr.kps_per_region[5] = 0x00;
fr.kps_per_region[6] = 0x00;
fr.kps_per_region[7] = 0x0b;
fr.kps_per_region[8] = 0x00;
fr.kps_per_region[9] = 0x00;
fr.kps_per_region[10] = 0x00;
fr.kps_per_region[11] = 0x00;
fr.kps_per_region[12] = 0x00;
fr.kps_per_region[13] = 0x00;
fr.kps_per_region[14] = 0x00;
fr.kps_per_region[15] = 0x02;
.. so on.

I have converted the ASCII file to binary, but I am at a loss of how to make the assignments happen correctly. I am struggling with the ordering, and also the use of an array of structs inside a (different) struct.

Here is what the binary file looks like:

00000000: ffff ffff ffff ffff 0064 000a 000a 000c  .........d......
00000010: 0200 0000 0000 0000 0b00 0011 1f0a 0503  ................
00000020: 0000 0000 0000 0200 0000 0000 0000 0000  ................
00000030: 0000 0000 0000 0000 0000 0000 0000 0000  ................
00000040: 0200 0000 0000 0000 0000 0204 0000 0000  ................
00000050: 0000 0000 0000 0000 0000 0000 0000 0000  ................
00000060: 0000 0000 0000 0000 0000 0000 0000 0000  ................
00000070: ffff ffff ffff ffff ffff ffff 0000 0000  ................
00000080: ffff ffff ffff ffff ffff ffff ffff ffff  ................
00000090: ffff ffff ffff ffff ffff ffff ffff ffff  ................
000000a0: ffff ffff ffff ffff ffff ffff ffff ffff  ................
000000b0: ffff ffff ffff ffff ffff ffff ffff ffff  ................
000000c0: ffff ffff ffff ffff ffff ffff ffff ffff  ................
000000d0: ffff ffff ffff ffff ffff ffff ffff ffff  ................

Here is what my attempt looks like:

union u_frame {
  char mem[sizeof(frame_t)];
  frame_t fr;
} uf;


void load_text_file(frame_t& f, const std::string& s)
{
  std::fstream input;
  input.open(s, std::fstream::in | std::fstream::binary);
  if(!input.is_open())
  {
    fprintf(stderr, "\nError opening file\n");
    exit(1); 
  }

  std::cout << "Size of frame_t = " << std::hex << sizeof(frame) << std::endl;
  std::cout << "File opened successfully\n";

  input.read(uf.mem, sizeof(frame_t));

  std::cout << "total_kps = " << uf.fr.total_kps
            << "\nnum_h_region = " << uf.fr.num_h_region
            << "\nnum_v_region = " << uf.fr.num_v_region
            << "\nnum_region = " << uf.fr.num_region;

  for(size_t i = 0; i < 8; ++i)
    std::cout << "\nreserved[" << i << "] = " << std::hex << (std::uint32_t)(uf.fr.reserved1[i]);

  // Printing the first 15 kp_per_region
  for(size_t i = 0; i < 15; ++i)
  {
    std::cout << "\nkps_per_region[" << i << "] = " << (std::uint32_t)(uf.fr.kp_per_region[i]); 
  }
}

int main() {

  frame_t left;

  std::cout << "Loading left frame -- \n";

  load_text_file(left, "line_ddr.bin");

  return 0;
}

This is the output:

Loading left frame -- 
Size of frame_t = 119600
File opened successfully
total_kps = ffff
num_h_region = ffff
num_v_region = ffff
num_region = ffff
reserved[0] = 0
reserved[1] = 64
reserved[2] = 0
reserved[3] = a
reserved[4] = 0
reserved[5] = a
reserved[6] = 0
reserved[7] = c
kps_per_region[0] = c
kps_per_region[1] = 2
kps_per_region[2] = 0
kps_per_region[3] = 0
kps_per_region[4] = 0
kps_per_region[5] = 0
kps_per_region[6] = 0
kps_per_region[7] = 0
kps_per_region[8] = 0
kps_per_region[9] = b
kps_per_region[a] = 0
kps_per_region[b] = 0
kps_per_region[c] = 11
kps_per_region[d] = 1f
kps_per_region[e] = a

Please let me know if additional information is needed (especially since I have pasted only a part of the input file).

Answer 1

Here is a Kaitai struct schema that processes the first 32 bytes of your file, you can check it in the Kaitai web IDE:

meta:
  id: frame_t
  file-extension: frame_t
  endian: be
seq:
  - size: 8
  - id: num_r
    type: u2
  - id: num_h_region
    type: u2
  - id: num_v_region
    type: u2
  - id: kps
    type: u2
  - id: kp_per_region
    type: u1
    repeat: expr
    repeat-expr: 16

Unfortunately I cannot help you write the rest as it is unclear to me how the rest of your file maps to the struct you showed.

Answer 2

Not getting into too much detail like how padding works and what are application limits - which are very useful to understand but not necessary to solve such an issue I would love to support a different approach.

We have two main problems to solve here:

1. we are serializing and deserializing data
2. we have a big file in which such data exists (or we want to write it)

As such we should ask 2 questions:

1. How can we serialize and deserialize data?
2. Can such a way of serializing and deserializing data work i.e. on file as a binary stream?

As an answer to the first question first library which comes to my mind would be Messagepack Now I need to answer the second one, and indeed documentation says it should be possible: Message pack memory management

Answer 3

Your code does exactly what you specified it to do. Takes an input stream and loads sizeof(frame_t) bytes into your uf. One thing to note is that your file/stream is read from "left to right", "byte by byte", so first two bytes (beacause you specified uint16_t) that you will load from ffff ffff ffff ffff 0064 000a 000a 000c ... into frame_t.kps will be ffff and not 000c. Good way to investigate further is to setup few breakpoints and debug your code step by step. You will learn important debugging technique and learn more about how your code works.

On a side note, I'm surprised you don't get stack overflow exception. That struct is huge.