CODEWITHSUNDEEP
go
fr33jumper
fr33jumper

Reputation: 103

upload large file - Error VirtualAlloc of x bytes failed with errno=1455 fatal error: out of memory

I have 10GB large file that I'm trying to upload with multipart/form-data in Go via Postman. Since I don't know much how file upload works in Go, I found tutorial on YouTube.

File upload works fine with smaller files, but on larger files always crashing with message: "runtime: VirtualAlloc of 9193373696 bytes failed with errno=1455 fatal error: out of memory". Here's the code I'm trying to make work:

    err := r.ParseMultipartForm(500 << 20)
    if err != nil {
        fmt.Fprintln(w, err)
    }

    file, handler, err := r.FormFile("file")

    if err != nil {
        fmt.Fprintln(w, err)
    }

    fmt.Fprintln(w, handler.Filename)
    fmt.Fprintln(w, handler.Size)
    fmt.Fprintln(w, handler.Header.Get("Content-type"))

    defer file.Close()

    saveLocation := "C:\\Users\\Pc\\go\\src\\github.com\\test\\uptest"
    tempFile, err := ioutil.TempFile(saveLocation, "upload")

    if err != nil {
        fmt.Fprintln(w, err)
    }

    defer tempFile.Close()

    fileBytes, err := ioutil.ReadAll(file)
    if err != nil {
        fmt.Fprintln(w, err)
    }
    tempFile.Write(fileBytes)

Upvotes: 1

Views: 1990

Answers (1)

novalagung
novalagung

Reputation: 11512

Using ParseMultipartForm() will require you to provide max memory allocation for temporarily store the uploaded file. If your file size is big (and it's bigger than the number of memory you allocated) then it's bad news for your memory resource.

From doc:

ParseMultipartForm parses a request body as multipart/form-data. The whole request body is parsed and up to a total of maxMemory bytes of its file parts are stored in memory, with the remainder stored on disk in temporary files. ParseMultipartForm calls ParseForm if necessary. After one call to ParseMultipartForm, subsequent calls have no effect.

Based on your error message, we can tell that the root cause of your issue is due to the uploaded file is larger than the memory you allocated, which is 500 << 20.

For handling big file upload, I suggest to take a look at MultipartReader() instead.

From doc:

MultipartReader returns a MIME multipart reader if this is a multipart/form-data or a multipart/mixed POST request, else returns nil and an error. Use this function instead of ParseMultipartForm to process the request body as a stream.

It's way faster approach and won't consume too much resource, it's because we will have the advantage of directly store the body (which is a stream data) into the destination file using io.Copy(), instead of writing it into temporarily storage first.

A simple usage example of MultipartReader():

reader, err := r.MultipartReader()
if err != nil {
    http.Error(w, err.Error(), http.StatusInternalServerError)
    return
}

for {
    part, err := reader.NextPart()
    if err == io.EOF {
        break
    }

    fmt.Println(part.FileName()) // prints file name
    fmt.Println(part.FormName()) // prints form key, in yor case it's "file"
    
    saveLocation := "C:\\Users\\Pc\\go\\src\\github.com\\test\\uptest"
    dst, err := os.Create(saveLocation)
    if dst != nil {
        defer dst.Close()
    }
    if err != nil {
        http.Error(w, err.Error(), http.StatusInternalServerError)
        return
    }

    if _, err := io.Copy(dst, part); err != nil {
        http.Error(w, err.Error(), http.StatusInternalServerError)
        return
    }
}

Reference: https://pkg.go.dev/net/http#Request.ParseMultipartForm

Upvotes: 5

Related Questions