CODEWITHSUNDEEP
arraysf#c#-to-f#gcallowverylargeobjects
Maslow
Maslow

Reputation: 18746

how do you create a large array?

Given 

// r is a System.Data.IDataRecord
var blob = new byte[(r.GetBytes(0, 0, null, 0, int.MaxValue))];
r.GetBytes(0, 0, blob, 0, blob.Length);

and r.GetBytes(...) returns Int64 Since Array.zeroCreate and Array.init take Int32 how do I create an empty array that is potentially larger than Int32.MaxValue?

Upvotes: 1

Views: 864

Answers (2)

Tomas Petricek
Tomas Petricek

Reputation: 243051

In .NET 4.5 and newer, the gcAllowVeryLargeObjects configuration element allows you to create individual objects that are larger than 2GB, but even that does not change the hard limit on number of elements in an array. To quote from the Remarks section:

The maximum number of elements in an array is UInt32.MaxValue.

The maximum index in any single dimension is 2,147,483,591 (0x7FFFFFC7) for byte arrays and arrays of single-byte structures, and 2,146,435,071 (0X7FEFFFFF) for other types.

I'm not entirely sure why UInt32 rather than Int32 (you can create array where the smallest index is Int32.MinValue, but I'm not sure you'll be able to do much with it :-)).

However, with the gcAllowVeryLargeObjects parameter, you can create very large arrays by creating an array of structs that contain multiple values - this allocates continuous memory block over 2GB:

[<Struct>]
type Quadruple<'T>(v1:'T, v2:'T, v3:'T, v4:'T) = 
  member x.V1 = v1
  member x.V2 = v2
  member x.V3 = v3
  member x.V4 = v4

let qa : Quadruple<byte>[] = Array.zeroCreate 10

I don't think this is particularly practical, but it is one possibility. In reality, it seems easier just to split your data into an array of arrays. One level of indirection will probably not cause too much overhead, when the individual arrays are around 2GB.

Upvotes: 6

Asik
Asik

Reputation: 22133

You can't. The maximum length of a single-dimensional array on .NET is System.Int32.MaxValue. Your C# code raises an OverflowException when the value is greater than this limit. The equivalent F# code is:

let blob =         
    let length = r.GetBytes(0, 0, null, 0, Int32.MaxValue)
    if length > int64(Int32.MaxValue) then
        raise (OverflowException())
    else
        Array.zeroCreate<byte>(int32(length))

Upvotes: 3

Related Questions