CODEWITHSUNDEEP
string-parsingflat-fileunivocity
santosh gamini
santosh gamini

Reputation: 81

Univocity - Parsing a fixedwidth flat file with one row - performance impact with 300 parallel threads

We have a project that deals with millions of transactions everyday which has some tight SLAs. As part of parsing the flat file that comes as input to a bean , we used beanio which was working better with out load. But with load its taking around 250ms to parse a flat file to a bean.

Requirement: Simple string has to converted to a single bean(nested and converted)

Heard the univocity can do better here - and tried the same with below settings.

FixedWidthParserSettings settings = new FixedWidthParserSettings();

    settings.getFormat().setLineSeparator("\n");
    settings.setRecordEndsOnNewline(false);
    settings.setHeaderExtractionEnabled(false);
    settings.setIgnoreLeadingWhitespaces(false);
    settings.setIgnoreTrailingWhitespaces(false);
    settings.setMaxColumns(100);
    settings.setMaxCharsPerColumn(100);
    settings.setNumberOfRecordsToRead(1);
    settings.setReadInputOnSeparateThread(false);
    settings.setInputBufferSize(10*1024);
    settings.setLineSeparatorDetectionEnabled(false);
    settings.setColumnReorderingEnabled(false);

When running with jmeter, with 200 parallel threads - the average time taken is 10ms(to parse and convert around 10 fields where in actual use case we have to the same for around 500 fields)

but when we increased it to 300 or 350 parallel threads , the average time was around 300ms. But our total SLA is around 10ms.

Any help here is highly appreciated!

Upvotes: 0

Views: 95

Answers (1)

Jeronimo Backes
Jeronimo Backes

Reputation: 6289

Probably you are running out of memory on your JVM. Try increasing it with the -Xms and -Xmx flags. Also too many threads won't help you if you don't have enough cores available.

Upvotes: 0

Related Questions