CODEWITHSUNDEEP
javacsvapache-drillunivocity
cgivre
cgivre

Reputation: 576

Can I speed up uniVocity CSV Parser?

I'm working on a project that is using the uniVocity CSV parser to read an InputStream which contains CSV data. Overall the CSV parser is working great, however there is one line which seems to be slow.

When I call the beginParsing() method, it seems to take almost 4 seconds just to execute this method in my unit test. Is there any way to speed this up? Thanks!

this.csvSettings = new CsvParserSettings();
csvSettings.setLineSeparatorDetectionEnabled(true);
RowListProcessor rowProcessor = new RowListProcessor();
csvSettings.setProcessor(rowProcessor);
csvSettings.setMaxCharsPerColumn(ValueVector.MAX_BUFFER_SIZE);
...
this.csvReader = new CsvParser(csvSettings);
logger.debug("Time to open CSV Parser: {} milliseconds", timer.elapsed().getNano() / 100000);

// This line is slow
csvReader.beginParsing(searchResults, "utf-8");
logger.debug("Time to open input stream: {} milliseconds", timer.elapsed().getNano() / 100000);

Here are the results:

19:35:23.475 [2112b444-e4af-cf7b-5006-7a79bbdae8dc:frag:0:0] DEBUG - Time to open CSV Parser: 262 milliseconds
19:35:24.912 [2112b444-e4af-cf7b-5006-7a79bbdae8dc:frag:0:0] DEBUG - Time to open input stream: 4634 milliseconds
19:35:24.921 [2112b444-e4af-cf7b-5006-7a79bbdae8dc:frag:0:0] DEBUG - Time to build schmea: 4727 milliseconds

Upvotes: 0

Views: 348

Answers (1)

Jeronimo Backes
Jeronimo Backes

Reputation: 6289

If ValueVector.MAX_BUFFER_SIZE is too large the process my have trouble allocating the required memory . For example, if it needs to start up with 2GB and the JVM is not initialized with enough ram (i.e. Xms < 2G) it will slow down because there's not enough memory available at startup.

Upvotes: 0

Related Questions