CODEWITHSUNDEEP
sqljsongrailscsv
Shenoy Tinny
Shenoy Tinny

Reputation: 1345

downloading huge files - application using grails

I am developing a RESTful web service that allows users to download data in csv and json formats that is dynamically retrieved from the database.

Right now I am using a StringWriter to write out the CSV data. My major concern is that the resultset could get very large depending the on the user input. In that case, having them all in memory doesn't seem to be a good idea to me.

I am thinking of creating a temp file, but how to make sure the file gets deleted soon after the download completes.

Is there a better way to do this.

Thanks for the help.

Upvotes: 0

Views: 343

Answers (4)

Chris
Chris

Reputation: 1236

If memory is the issue, you could simply write out to the response writer that writes directly to the output stream? This way you're not storing anything (much) in memory and no need to write out temporary files:

// controller action for CSV download
def download = {
    response.setContentType("text/csv")
    response.setHeader("Content-disposition", "attachment;filename=downloadFile.csv")
    def results = // get all your results
    results.each { result ->
        out << result.col1 << ',' << result.col2 // etc
        out << '\n'
    }
}

This writes out to the output stream as it is looping round your results.

In theory You can make this even more memory efficient by using a scrollable results set - see "Using Scrollable Results" section of Querying with GORM - Criteria - and looping round that whilst writing out to the response writer. In theory this means you're also not loading all your DB results into memory, but in practice this may not work as expected if you're using MySQL (and its Java connector). Manually batching up queries may work too (get DB rows 1-10000, write out, get 10001-20001, etc)

This kind of thing might be more difficult with JSON, depending on what library you're using to render your objects.

Upvotes: 1

deluan
deluan

Reputation: 1865

To create a temp file that gets automatically deleted after the session has expired, you can use the Session Temp Files plugin.

Upvotes: 0

Greg
Greg

Reputation: 116

Depending on the number of files involved you can always use http://download.oracle.com/javase/1,5.0/docs/api/java/io/File.html#deleteOnExit() to ensure the file is blown away when the VM shuts down.

Upvotes: 0

Kyle
Kyle

Reputation: 887

Well, the simplest solution to preventing temp files from sticking around too long would be a cron job that simply deletes any file in the temp directory that has a modified time older than, say, 1 hour.

If you want it to all be done within Grails, you could design a Quartz job to clean up files. This job could either do as described above (and simply check modification timestamps to decide what to delete) or you could run the job only "on demand" with a parameter of a file name to be deleted. Once the download action is called you could schedule the cleanup of that specific file for X minutes later (to allow enough time for a successful download). The job would then be in charge of simply deleting the file.

Upvotes: 0

Related Questions