CODEWITHSUNDEEP
c++vectorvisual-studio-2017sgxenclave
落水湮花
落水湮花

Reputation: 3

How much memory(MB) can the vector variable occupy in enclave of Intel sgx?

I want to immigrate PageRank algorithm in the sgx enclave. The algorithm uses vector to save the edge relationship and matrix.

vector<size_t> num_outgoing; // number of outgoing links per column
vector< vector<size_t> > rows; // the rowns of the hyperlink matrix
map<string, size_t> nodes_to_idx; // mapping from string node IDs to numeric
map<size_t, string> idx_to_nodes; // mapping from numeric node IDs to string
vector<double> pr; // the pagerank table

The application runs well when it stores less than 9000 edges. Once it increases to 10000 edges or more, the application crushes, and throws out an unhandled exceptionenter image description here . I also run the same code outside the enclave, it runs well when it stores 90000 edges.

By debugging, I found the application fails at the places below.

if (rows.size() <= max_dim) {
    max_dim = max_dim + 1;
    rows.resize(max_dim);
    if (num_outgoing.size() <= max_dim) {
        num_outgoing.resize(max_dim);
    }
}

However, the variable 'rows' cannot be resized larger once it owns 13896 element. I'm confused that 'rows' only occupies about 300kb and 'num_outgoing' only occupies about 100kb. It's far less than the allow size. There is 128MB space in total for the enclave application.

My enclave config file is listed as follows. I try to change the value of StackMaxSize, however, it seems useless.

<EnclaveConfiguration>
    <ProdID>0</ProdID>
    <ISVSVN>0</ISVSVN>
    <StackMaxSize>0x400000</StackMaxSize>
    <HeapMaxSize>0x100000</HeapMaxSize>
    <TCSNum>1</TCSNum>
    <TCSPolicy>1</TCSPolicy>
    <DisableDebug>0</DisableDebug>
    <MiscSelect>0</MiscSelect>
    <MiscMask>0xFFFFFFFF</MiscMask>
    <EnableKSS>0</EnableKSS>
    <ISVEXTPRODID_H>0</ISVEXTPRODID_H>
    <ISVEXTPRODID_L>0</ISVEXTPRODID_L>
    <ISVFAMILYID_H>0</ISVFAMILYID_H>
    <ISVFAMILYID_L>0</ISVFAMILYID_L>
</EnclaveConfiguration>

The format of the input edge is shown as below. The first number is the "from" node, the second number is the "to" node

1 0
2 0
3 1
4 3
5 4
6 0
7 1
8 0
9 1
10 0

I wonder how to config enclave to make it allow bigger vector variable? The problem exists on both win10 and ubuntu.

Upvotes: 0

Views: 573

Answers (3)

X99
X99

Reputation: 915

Altough the other answers are perfectly correct, Intel recently increased the memory limit to up to 1Tb, see here for more info.

Upvotes: 0

kaichi
kaichi

Reputation: 101

All seems like you are running out of memory. The memory limits (stack and heap) are set in the config file using StackMaxSize and HeapMaxSize (see Developer Reference for details). The size of the EPC (128MB or 256MB) has nothing to do with it. Here, you are not bounded by the EPC size but by the heap and stack.

Increasing the stack size does not change anything because dynamically allocated memory resides on heap. In turn, you should be looking at the max heap size. Currently, you have it set to 0x100000 (=1MB), which is quickly used up by your data. Try increasing it and see if you can accommodate larger vectors.

I think on Windows your enclave has to entirely fit into EPC but on Linux you can create enclaves with tens of GBs. Be aware that once your (Linux) enclave memory usage exceeds ~90 MB you will start noticing EPC paging and, with it, a huge performance degradation.

Upvotes: 1

exfalso
exfalso

Reputation: 26

SGX CPUs (before Icelake) have a limited EPC, this is 128M for CPUs like Skylake, but you can also get 256M with Xeon E-2200. This does not mean that your application cannot use more memory, it simply means that the hardware-accelerated memory range is limited. Pages that don't fit into the EPC are swapped to non-EPC memory (at a considerable performance cost), however this is only implemented in the linux driver.

So, you can set the enclave heap to something much larger like 2G. What you'll see is slower startup time (that 2G must be completely initialized), and if your compute's memory access pattern is scattered in that 2G range then you'll see extremely degraded performance. So try to keep your access patterns local, use sequential/scanning like operations etc, the usual considerations for cache-friendly compute.

Regarding your actual issue, it could be that you're running out of the allocated heap, and that vector just happens to be the "last straw". Remember that the heap must contain not only these datastructures but also the code itself. If you're parsing the input from some serialized format then it could be that the serialized bytes are still retained in memory, if you have other state then that also uses memory, there can be many sources of extraneous usage. If you're using the Intel SDK then I'd recommend compiling in simulation mode, or just link your application into a non-SGX ELF and use usual memory debugging tools to track memory usage.

Upvotes: 0

Related Questions