CODEWITHSUNDEEP
javaperformancelarge-data
user3015565
user3015565

Reputation: 413

How to handle large amounts of data?

I have a project for school where I was given 103MB of data (Water Stream Quality Results in Ontario) which equates to 200 000+ lines of data. I need to be able to take in the data and create a visual out of it (e.g. a map of Ontario with dots representing where X chemical was found in 20XX).

I have tried taking in everything more or less all at once which my computer obviously couldn't handle. I was wondering if there is a strategy or function that can help me with this.

I would preferable like to have the information in a 2D ArrayList after the data has been taken in.

Example

Array Name: rawData

[0], [1], [2]

2002, 2003, 2004

station num, station num, station num

chem. found, chem. found, chem. found

etc.

If this is an inefficient way to have the data stored, what would be a better way?

Any help would be appreciated.

Side Note: the data has been given to me over 10+ files. Also, the file extension is a .csv so each row is separated by a comma (e.g. [year, chemical found, concentration percent>, etc.])

Upvotes: 0

Views: 527

Answers (2)

Fallso
Fallso

Reputation: 1301

In addition to Ardash's answer, you will probably need to read in the data in chunks. If you use a BufferedReader, you can read CSV files line by line.

Upvotes: 0

Adarsh
Adarsh

Reputation: 3641

You can use ArrayLists. Create a class with the values you need to store in the objects.

For eg :

class Data{
  String Year;
  String found;
  ---one property for each of the value you need--
}

You can then populate the objects of this class and add them to the arrayList.

Upvotes: 2

Related Questions