CODEWITHSUNDEEP
c++linuxperformancememoryprocess
Wallace
Wallace

Reputation: 651

Why this app doesn't consume as much memory as expected

I wrote a simple application to test memory consumption. In this test application, I created four processes to continually consume memory, those processes won't release the memory unless the process exits.

I expected this test application to consume the most memory of RAM and cause the other application to slow down or crash. But the result is not the same as expected. Below is the code:

 #include <stdio.h>
 #include <unistd.h>
 #include <list>
 #include <vector>

 using namespace std;
 unsigned short calcrc(unsigned char *ptr, int count)
 {
     unsigned short crc;
     unsigned char i;

     //high cpu-consumption code 
     //implements the CRC algorithm
     //CRC is Cyclic Redundancy Code
 }


 void* ForkChild(void* param){
    vector<unsigned char*>  MemoryVector;
    pid_t PID = fork();
    if (PID > 0){
        const int TEN_MEGA = 10 * 10 * 1024 * 1024;
        unsigned char* buffer = NULL;
        while(1){
            buffer  = NULL;
            buffer = new unsigned char [TEN_MEGA];
            if (buffer){
                 try{
                    calcrc(buffer, TEN_MEGA);
                    MemoryVector.push_back(buffer);
                 } catch(...){
                    printf("An error was throwed, but caught by our app!\n");
                    delete [] buffer;
                    buffer = NULL;
                 }
            }
            else{
                 printf("no memory to allocate!\n");
                 try{
                     if (MemoryVector.size()){
                        buffer = MemoryVector[0];
                        calcrc(buffer, TEN_MEGA);
                        buffer = NULL;
                     } else {
                        printf("no memory ever allocated for this Process!\n");
                        continue;
                     }
                 } catch(...){
                    printf("An error was throwed -- branch 2," 
                           "but caught by our app!\n");
                    buffer = NULL;
                 }
             }
         }  //while(1)
    } else if (PID == 0){
    } else {
      perror("fork error");
    }   

    return NULL;
}


int main(){
int children = 4;
    while(--children >= 0){
    ForkChild(NULL);
    };

    while(1) sleep(1);
    printf("exiting main process\n");
    return 0;
 }

  1. TOP command 

    PID  USER      PR  NI  VIRT  RES  SHR S  %CPU %MEM    TIME+  COMMAND           
2775 steve     20   0 1503m  508  312 R  99.5  0.0   1:00.46 test              
2777 steve     20   0 1503m  508  312 R  96.9  0.0   1:00.54 test              
2774 steve     20   0 1503m  904  708 R  96.6  0.0   0:59.92 test              
2776 steve     20   0 1503m  508  312 R  96.2  0.0   1:00.57 test

    Though CPU is high, but memory percent remains 0.0. How can it be possible??

  2. Free command 

                      free  shared    buffers     cached          
Mem:           3083796       0      55996     428296

    Free memory is more than 3G out of 4G RAM.

Does there anybody know why this test app just doesn't work as expected?

Upvotes: 3

Views: 291

Answers (5)

Ryan
Ryan

Reputation: 734

If you want to gobble up a lot of memory:

int mb = 0;
char* buffer;
while (1) {
    buffer = malloc(1024*1024);
    memset(buffer, 0, 1024*1024);
    mb++;
}

I used something like this to make sure the file buffer cache was empty when taking some file I/O timing measurements.

As other answers have already mentioned, your code doesn't ever write to the buffer after allocating it. Here memset is used to write to the buffer.

Upvotes: -1

smac89
smac89

Reputation: 43128

But your program is infact doing what you expected it to do. As an answer has pointed out (@ Michael Foukarakis's answer), memory not used is not allocated. In your output of the top program, I noticed that the column virt had a large amount of memory on it for each process running your program. A little googling later, I saw what this was:

VIRT -- Virtual Memory Size (KiB). The total amount of virtual memory used by the task. It includes all code, data and shared libraries plus pages that have been swapped out and pages that have been mapped but not used.

So as you can see, your program does in fact generate memory for itself, but in the form of pages and stored as virtual memory. And I think that is a smart thing to do

A snippet from this wiki page

A page, memory page, or virtual page -- a fixed-length contiguous block of virtual memory, and it is the smallest unit of data for the following:

  • memory allocation performed by the operating system for a program; and
  • transfer between main memory and any other auxiliary store, such as a hard disk drive.

...Thus a program can address more (virtual) RAM than physically exists in the computer. Virtual memory is a scheme that gives users the illusion of working with a large block of contiguous memory space (perhaps even larger than real memory), when in actuality most of their work is on auxiliary storage (disk). Fixed-size blocks (pages) or variable-size blocks of the job are read into main memory as needed.

Sources:

Upvotes: 0

user703016
user703016

Reputation: 37945

Linux uses optimistic memory allocation: it will not physically allocate a page of memory until that page is actually written to. For that reason, you can allocate much more memory than what is available, without increasing memory consumption by the system.

If you want to force the system to allocate (commit) a physical page , then you have to write to it.

The following line does not issue any write, as it is default-initialization of unsigned char, which is a no-op:

buffer = new unsigned char [TEN_MEGA];

If you want to force a commit, use zero-initialization:

buffer = new unsigned char [TEN_MEGA]();

Upvotes: 6

Sebastian Redl
Sebastian Redl

Reputation: 71989

fork() returns the PID in the parent, and 0 in the child. Your ForkChild as written will execute all the work in the parent, not the child.

And the standard new operator will never return null; it will throw if it fails to allocate memory (but due to overcommit it won't actually do that either in Linux). This means your test of buffer after the allocation is meaningless: it will always either take the first branch or never reach the test. If you want a null return, you need to write new (std::nothrow) .... Include <new> for that to work.

Upvotes: 1

Michael F
Michael F

Reputation: 40829

To make the comments into an answer:

  • Linux will not allocate memory pages for a process until it writes to them (copy-on-write).
  • Additionally, you are not writing to your buffer anywhere, as the default constructor for unsigned char does not perform any initializations, and new[] default-initializes all items.

Upvotes: 3

Related Questions