CODEWITHSUNDEEP
csvfilemapreducenetsuitesuitescript2.0
Maira S
Maira S

Reputation: 47

How to create multiple CSV files to avoid 10MB file content limit using map reduce script?

I hope you are welll. I am trying to store dataset results in a csv file. but I am getting error as, SSS_FILE_CONTENT_SIZE_EXCEEDED error. So my plan is to create multiple files. But, I need to check file size, if its 10MB then create second file and store subsequent data in that. My concern is, before saving the file we couldn't get the file size. Thanks in advance!

function summarize(summary) {
                let csvContent = [];
                let datasetObj = dataset.load({ id: 'custdataset70' }); 
                let headers = datasetObj.columns.map(col => col.label || col.id);
                csvContent.push(headers);
    
                summary.output.iterator().each((key, value) => {
                    csvContent.push(value); // Keep value as string
                    return true;
                });
    
                if (csvContent.length <= 1) { // Only headers, no data
                    log.debug('No Data', 'No unique dataset records found.');
                    return;
                }
    
                let fileId = saveCSVFile(csvContent.join('\n'));
                if (fileId) {
                    sendEmail(fileId);
                }
                          
       
        }
        function saveCSVFile(csvContent) {
                let today = format.format({ value: new Date(), type: format.Type.DATE });
                let csvFile = file.create({
                    name: 'DatasetResults_' + today + '.csv', // Dynamic filename
                    fileType: file.Type.CSV,
                    contents: csvContent,
                    folder: 5081389 
                });
                return csvFile.save();
                }

Upvotes: 0

Views: 27

Answers (1)

Arek Kubiński
Arek Kubiński

Reputation: 269

You can check how much space takes up empty file (since it seems like you are putting some data inside by specifying file type), then just take you array I'm guessing line by line and add the value of new Blob(["myString"]).size, this should output the size in bytes for the line of data you are putting inside csv.

Then if the size is bigger or equal to threshold just put it inside new file.

Upvotes: 0

Related Questions