CODEWITHSUNDEEP
c#unity-game-enginesplitstring-parsing
Splash
Splash

Reputation: 1288

String read and parsing performance with C# and Unity

Background

First, I have a text file (CSV) with a few columns and hundred thousands of rows. The total file size is 10MB. I used resource with Unity so it loads as Text

The related code is:

TextAsset txtData = Resources.Load("data.csv") as TextAsset;
string txt = txtData.text;
strReader = new StringReader(txt);
string line0 = strReader.ReadLine();

....

currentLine =strReader.ReadLine();
while (true) {// if last line is nothing we quit
    var values = currentLine.Split(',');    
    try {
        float x = (float)Convert.ToSingle(values[colX]);
        float y = (float)Convert.ToSingle(values[colY]);
        float z = (float)Convert.ToSingle(values[colZ]);
        float w = (float)Convert.ToSingle(values[colSize]);
        runningList.Add(v1);
    }catch(Exceptoion e){
    }
    currentLine = strReader.ReadLine();
}

Problem

It was found that the reading plus parsing is slow so that it affects the Unity visual effect. So I used log file to see. I count time for every 500 rows. Strange enough, the last group takes 12ms (500 rows), the second from last takes 20ms, the time is linearly increasing to 1.5-1.7 seconds for the first group.

More Info

When Unity is drawing at 90 Hz, I am using a thread to read the string and parse the data.

Question

Where should I look for problems? I used Unity resource, string reader, split, parsing to float. Where is the cause and is there a way to improve?

It looks strange as the time reduces.

Update

after I used file stream reader, it is 2ms each group. So it is Unity TextAsset?

Upvotes: 2

Views: 2316

Answers (1)

OLNG
OLNG

Reputation: 185

Given the behavior, it's almost the C# bug reading Linux text files.

C# expects \r\n but Linux has only \n. Then it is reasonable that each line read will go through the whole file and found out it is Linux file and the time to parse lines will be proportional to the remaining file size

Upvotes: 1

Related Questions