CODEWITHSUNDEEP
c
jack
jack

Reputation: 75

Allocate more than 4GB memory in C

I need to allocate more than 4GB memory in my C program (running on Windows - 64 bit).
Obviously using just malloc does not solve the problem.

I've read some of the previous posts in this manner, and it seems like VirtualAlloc may solve my issue.
I tried to use it, and I don't understand all the parameters associated with this function. I think I need to define exactly the address in which the memory should be allocated, but I don't know how to do it.
Is there a simple information about it, or a simple example that I can use?

I also do not need to allocate all in one chunk, so it might be easier. Any alternative way to allocate more than 4gb memory will also be accepted.

Thanks.

Upvotes: 4

Views: 7948

Answers (4)

chux
chux

Reputation: 153338

Assuming 1) size_t/SIZE_MAX is smaller than available memory 2) and OP needs to perform a single large allocation:

Test your system's calloc().

Depending on implementation, calloc() can allocate more than SIZE_MAX bytes. The below will allocate 4*SIZE_MAX bytes.

uint32_t *p;
size_t nmemb = SIZE_MAX;
p = calloc(nmemb, sizeof *p);
asset(p);

for (size_t i = 0; i < nmemb; i++) {
  p[i] = ...
}

free(p);

I've used this type of code when size_t was 16-bit and available memory was many times larger than that.

OTOH, somecalloc(size_t nmemb, size_t size) will not allow an allocation where (nmemb * size) > SIZE_MAX. YMMV

Upvotes: 1

J. Toran
J. Toran

Reputation: 177

The easiest way to allocate more than 4GB of memory in Windows is to just compile as a 64-bit program. If you're using Visual Studio there are 2 ways to do that.

On the initial screen, at the top next to the "debug" dropdown there is a dropdown to specify what mode you want the program to compile to. The default is "x86". Click the dropdown and select "x64".

The second way is to click on the properties of the project. On the next screen, next to "platform" is the Configuration Manager button. Click it and the Configuration Manager screen displays. On the "Active solution platform" click the dropdown and select "x64".

Then the normal malloc will work with more than 4GB.

This is using Visual Studio 2015. I assume you can do similar in other versions of VS.

Upvotes: 1

Peter
Peter

Reputation: 36597

Contrary to your statement, malloc() will solve the problem, assuming you are using an implementation (which includes the compiler and library) that can produce a 64-bit executable, and have configured it to do so (i.e. to build a 64-bit target). Some toolchains are capable of building 64-bit targets but by default (e.g. as used in an associated IDE) will only produce a 32-bit executable.

If you are building a 32-bit target (i.e. producing a 32-bit executable) then, yes, malloc() will be limited to 4GB. A 32-bit executable can be executed on a 64-bit system (assuming an operating system which permits that). However, the program's use of malloc() will still be limited to 4GB in that case.

Which means you need to ensure you have a compiler that can build 64 bit programs AND use it to build a 64-bit target.

Of course, another question you should ask is whether you really need to allocate more than 4GB in a single chunk. While there are circumstances where that is appropriate, more often than not, a program which needs to do that is a sign of poor or lazy design.

Upvotes: 8

Art
Art

Reputation: 20392

Obviously using just malloc does not solve the problem.

Why? size_t should be large enough to cover all the address space your program has accessible, malloc takes a size_t and there are no standard reasons for why it shouldn't work.

I can unfortunately not try this on a Windows machine right now, but on a Linux machine this program works just fine (to show that there are no limitations in C that would prevent you from doing this):

#include <stdlib.h>
#include <stdio.h>
#include <string.h>

int
main(int argc, char **argv)
{
    size_t sz = 40000000000;
    char *a = malloc(sz);

    memset(a, 'a', sz);

    printf("%.4s", &a[sz-5]);

    return 0;
}

I allocate 40GB, fill it with the letter 'a' and then access it just to see that things worked. And they do. (Warning: don't run this program on a machine that doesn't have this much memory, systems can get quite unhappy when you overcommit memory).

Does this program work for you (size adjusted to what your hardware can do, obviously)? If yes, then you might have a problem with prototypes or data types for the size rather than a problem with malloc.

Upvotes: 5

Related Questions