CODEWITHSUNDEEP
c#asp.netvisual-studiocsv
Player 2nd
Player 2nd

Reputation: 251

System.ArgumentOutOfRangeException: 'capacity was less than the current size. Parameter name: requiredLength'

I need to generate a random alphanumeric of length between 1k bytes to 2K bytes to store it to the csv

I got an error

System.ArgumentOutOfRangeException: 'capacity was less than the current size. Parameter name: requiredLength'

    private const String chars = " abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789";
    protected void Button1_Click(object sender, EventArgs e)
    {
        String ID;
        String stringContent;
        int records = 100000;
        StringBuilder csvcontent = new StringBuilder();

        for (int r = 0; r < records; r++)
        {
            Guid guid = Guid.NewGuid();
            ID = guid.ToString();
            stringContent = GenerateRandomString(256000);
            csvcontent.AppendLine(ID + "," + stringContent);
        }

        File.AppendAllText(csvPath, csvcontent.ToString());
    }

    File.AppendAllText(csvPath, csvcontent.ToString());

    private string GenerateRandomString(int strLength)
    {
        return new string(Enumerable.Repeat(chars, strLength)
          .Select(s => s[random.Next(s.Length)]).ToArray());
    }

Is it because the string can't hold that much characters ? Is there any way to fix it ?

Upvotes: 1

Views: 5789

Answers (3)

kessmahat
kessmahat

Reputation: 1

As @JesseChunn already mentioned, I think you should check why you generate such huge strings. If I am not wrong you are trying to generate nearly 24 GB of random string data, correct?

But if you got still the need to create a big amount of string data, you could write the random strings directly to the file.

e.g.

private const String chars = " abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789";
protected void Button1_Click(object sender, EventArgs e)
{
    String ID;
    byte[] stringContent;
    int records = 100000;

    using(var textWriter = File.AppendText(csvPath))
    {
        for (int r = 0; r < records; r++)
        {
            Guid guid = Guid.NewGuid();
            ID = guid.ToString();
            stringContent = GenerateRandomString(256000);
            textWriter.WriteLine($"{ID}, {stringContent}");
         }
    }      
}

private string GenerateRandomString(int strLength)
{
    return new string(Enumerable.Repeat(chars, strLength)
      .Select(s => s[random.Next(s.Length)]).ToArray());
}

Upvotes: 0

Lasse V. Karlsen
Lasse V. Karlsen

Reputation: 391476

Let's do the math here.

You want to write 100.000 rows, with this format:

<guid>,<256.000 characters>

Let's ignore line feed character(s) at the end just for simplicity. The above line is 36+1+256.000 characters long, 256.037 in total. Multiply this by 100.000 rows, and you get a total number of characters of 25.603.700.000 characters. Since a character in memory is stored as a 16-bit unicode codepoint, that means double that in terms of bytes, or 51.207.400.000 bytes, which is roughly 47.7GB of memory.

In one StringBuilder.

This is simply not possible. A StringBuilder has a limit of somehwere around 2.1 million characters, and you want to store 25 billion characters in it.

It's simply not possible, and even if you managed to find a way around it, or the limit for a StringBuilder was changed to handle this, storing this in memory only for then afterwards to dump it to a file is unnecessary. Instead just write it directly to your file to begin with. Here is a literal translation of your code, which can further be simplified somewhat.

protected void Button1_Click(object sender, EventArgs e)
{
    int records = 100000;

    File.AppendAllLines(csvPath,
        (from r in Enumerable.Range(0, records)
         let guid = Guid.NewGuid()
         let stringContent = GenerateRandomString(256000)
         select $"{guid},{stringContent}"));
}

Note that this will add around 27-28GB to the file. I question the usage you could have for a text file of this size, at this point I would probably start to think about a database, because searching for a Guid in this big file is going to have very low performance.

Upvotes: 3

JesseChunn
JesseChunn

Reputation: 595

The StringBuilder in .Net has limits. Check the value of:

csvcontent.MaxCapacity;

You will probably get 2,147,483,647 (which is Int32.MaxValue). I say "probably" because Microsoft says that could change in future implementations.

You cannot exceed the size returned from that call (in fact it is slightly smaller due to overhead).

You will need to either break your string into multiple StringBuilders or, better yet, figure out why you are creating such a large string and change your design to not do that, since it is unusual to ever need such a huge string, practically speaking.

Why not just append the data to the file directly inside the loop instead of using the intermediate stringbuilder?

Upvotes: 1

Related Questions