CODEWITHSUNDEEP
javacsvexport-to-csvsupercsv
Leejoy
Leejoy

Reputation: 1446

Issue while working with SuperCSV parser

I have created a web application module to import CSV file to DB. After the import process the user will be displayed the summary like, the total number of records in input file, the total number of good records that got imported to db and the total number of rejected records. The user can download the success and error records to verify.

The issue is some duplicate records are appended to success and error files. For example the user gets a message as follows:

Total number of records: 2099
Number of good records: 1694
Number of skipped records: 405
Click to download success file: Import_20121012184828_success.csv
Click to download error file: Import_20121012184828_errors.csv

When we check the db the number of records in the table is exactly '1694' (Same as number of good records as expected.). But in Import_20121012184828_success.csv file there are '1741' records. But in eclipse console it prints same as in db i.e after the last success record there is no duplication. So once exited from the loop and when I call successCsvMapWriter.close(); and errorCsvMapWriter.close(); some where from buffer the records are getting added up. I cant figure this out.

Note: For small number of inputs I din't notice any issues at all. Tried with versions 1.52 and 2.0.0 beta.

Following is the code to create the resultant files:

File successFile = new File(csvFileDetails.getSuccessFileUrl());
File errorFile = new File(csvFileDetails.getErrorFileUrl());
// Create result files.
successFile.createNewFile();
errorFile.createNewFile();

String[] header = csvFileDetails.getHeader();
String[] errorFileHeader = ArrayUtils.add(header,
    CatalogImportConstanst.CSV_ERROR_HEADER);

// Also tried CsvPreference.STANDARD_PREFERENCE 
ICsvMapWriter successCsvMapWriter = new CsvMapWriter(new BufferedWriter(
    new FileWriter(successFile)), CsvPreference.EXCEL_PREFERENCE);
ICsvMapWriter errorCsvMapWriter = new CsvMapWriter(new BufferedWriter(
    new FileWriter(errorFile)), CsvPreference.EXCEL_PREFERENCE);

successCsvMapWriter.writeHeader(header);
successCsvMapWriter.flush();
errorCsvMapWriter.writeHeader(errorFileHeader);
errorCsvMapWriter.flush();

int errorCount = 0;
int successCount = 0;
for (Map<String, String> csvRecord : csvAsList) {
    if (csvRecord.get(CatalogImportConstanst.CSV_ERROR_HEADER) != null
        && csvRecord.get(CatalogImportConstanst.CSV_ERROR_HEADER).trim().length() > 0) {
        errorCsvMapWriter.write(csvRecord, errorFileHeader);
        errorCsvMapWriter.flush();
        errorCount++;
        System.out.println("Error record: "+ csvRecord);
    } else {
        successCsvMapWriter.write(csvRecord, header);
        successCsvMapWriter.flush();
        successCount++;
        System.out.println("Success record: "+ csvRecord);
    }
  }
successCsvMapWriter.close();
errorCsvMapWriter.close();

Thanks in advance.

Upvotes: 0

Views: 646

Answers (1)

James Bassett
James Bassett

Reputation: 9918

Just to clarify, when you say that your success file has 1741 records are you counting CSV records or the number of lines in the file? It could be that you have records with embedded newlines, which mean you can't simply count the number of lines (i.e. is this actually duplication or are you counting incorrectly?).

Instead of maintaining the record count yourself - you can use getRowNumber(), which returns the number of rows written (including the header). If you call this at the end of your method, what does it return?

successCsvMapWriter.close();
errorCsvMapWriter.close();

System.out.println("Success records (including header): " +
    successCsvMapWriter.getRowNumber());
System.out.println("Error records (including header): " +
    errorCsvMapWriter.getRowNumber());

Oh, and you don't need all of those flush() statements - the streams will be flushed when you call close().

Upvotes: 2

Related Questions