CODEWITHSUNDEEP
c#asp.net
Dov95
Dov95

Reputation: 85

In .net reading large file it reads only 1000 rows

I have a file with products and prices, I need to write then to database, however I am trying to read a file and it reads only 1000 rows, and it says it is end of file although file has over 120000 rows. Look like reader is reading some rows from start of the document then some random from the middle and then some from the end of the file. Even if i do not write them to database and only writing them to database I got same result. Here is my code:

public async Task LoadProductsFromExcel()
    {
        var file = @"F:\Links\productsToImport.csv";
       var fileStream = new FileStream(file, FileMode.Open, FileAccess.Read);
        using (var streamReader = new StreamReader(fileStream))
        {   
            while (!streamReader.EndOfStream)
            {
                var line = streamReader.ReadLine();
                var data = line.Split(new[] { ';' });
                var product = new Product() { Name = data[1], Code = data[0]};       
                context.Products.Add(product);
                Console.WriteLine(data[0]+"   "+ data[1]);

            }

        }
        await unitOfWork.CompleteAsync();
    }

Is there a problem with buffer of stream or some any other problem? Or maybe I am reading a file wrong.

Upvotes: 0

Views: 609

Answers (1)

Kevin
Kevin

Reputation: 2243

I can see a few possibilities.

First, the .csv file might not have Newline chars separating all the records, or it might have strange characters that screw up the ReadLine() loop. Open the file with Notepad++ (and make sure to show all symbols) or throw it into a Hex Editor.

Second, you've got a bug in your code that could generate an exception - it may be that this code is actually getting 1000-lines in and is exiting out due to an exception that's silently caught in a try/catch block higher up:

            var line = streamReader.ReadLine();
            var data = line.Split(new[] { ';' });
            var product = new Product() { Name = data[1], Code = data[0]};       
            context.Products.Add(product);
            Console.WriteLine(data[0]+"   "+ data[1]);

... if the line= gets reads a line that doesn't contain a semicolon, it's going to make data= a 1-element array. Then the next line, with Name = data[1] is going to generate an 'Past the end of an array' exception. So another possibility is that your source file has a line without a semi-colon and you've got a try/catch block that's catching it.

Upvotes: 1

Related Questions