CODEWITHSUNDEEP
c++large-data
user51957
user51957

Reputation: 21

Long compilation for an initialization of large vector of vectors

I have a very large constant 2-dimension array to store in a C++ file. Dimensions are about 1,000,000 rows and 2 to 10 elements in each row, with uint8_t as elements.

The number of elements in each row being variable, I wanted to use vectors for the rows, and possibly vector of vectors for the full array, so I tried 3 ways for the initialization 

const uint8_t array[1000000][10] = {{...},...,{...}};              /* Case 1 */
const std::vector<uint8_t> array[1000000] = {{...},...,{...}};     /* Case 3 */
const std::vector<std::vector<uint8_t>> array = {{...},...,{...}}; /* Case 2 */

The compilation using g++ is very different between the three cases.

Case 1 compiles fast, but I fear that some space is wasted to keep the alignment? Is this different for Cases 2 and 3?

Case 2 and Case 3 take a very long time to compile. Adding -ftime-report shows that the g++ "phase opt and generate" step is taking a very long time.

Any idea what to change in the code and/or to make the compilation faster?

Upvotes: 2

Views: 271

Answers (1)

align
align

Reputation: 361

You can write the 'whole data' in a separate assembly file, and link that file and c++ sources into single binary. This will be extremely fast compared to your case.

Sample asembly file:

 .globl large_data
    .section .rodata
    .p2align 5
    .type large_data, "object"
    .size large_data, <insert_size_here>
large_data:

    .byte 77,90,144,0,3,0,0,0,4,0,0,0,255,2..... and so on

Sample C++ code:

extern const uint8_t large_data [];

If you link these source together, you can directly access to data, for e.g:

uint8_t x = large_data[0];

In this way, you would need to compile assembly file only once.

Upvotes: 3

Related Questions