CODEWITHSUNDEEP
csvgrailsbigdata
janDro
janDro

Reputation: 1466

Grails App with Huge Tables

I'm trying to create a database from existing csv files that are about 20,000 columns wide and 700 rows deep. In grails I would like the 20,000 column domain to belongTo another simpler domain (about 200 columns). But upon compilation I get: 

java.lang.RuntimeException: Class file too large!

Which is understandable because it's way too much data. My question is, what is the best approach to handle this problem in grails? Should I simply break up the big table into separate domains? Look for a different table format?

I'm specifically worried about:

1) Search time, parsing search methods then delegating to sub domains.

2) Importing the data from the huge csv file into the domains.

Upvotes: 2

Views: 457

Answers (1)

Burt Beckwith
Burt Beckwith

Reputation: 75681

When you crash into a JVM size limit like this, take it as a big hint that your approach is way off. As I mentioned in another question earlier this week, we shouldn't even know what these limits are, much less be anywhere near hitting them.

I don't see much benefit in using something like GORM or even an O-O approach in general to this much data. It's not an object in any realistic, usable sense - it's a massive bunch of data. You'll need to programmatically access everything anyway even if it did work, since hand-managing the code for that would be crazy amounts of code. Do you really plan on creating one or more instances of these beasts and passing them around as method args?

You'll need to look at this from a big data perspective, not an ORM perspective.

Upvotes: 8

Related Questions