Reputation: 1838
Disclaimer: I'm not sure if the title is my actual problem, but it's the only reason that makes sense for what's below.
Scenario
I am uploading large videos to my Golang server. I have an 8GB file that takes roughly 15 minutes locally and works well. However, on my staging server (Google Cloud) this file takes a little over an hour to upload and this is where the problem hits.
Problem
My client is waiting on the file to finish uploading in order to call another API endpoint, but only on the staging server does it fail. So it seems like when my server is looking to write the response, then the connection won't allow it because the client shows a 502
error and a CORS error, which I know is Chrome's way of saying "We don't know, check your server".
Code
Inside my SrcHandler
is this:
defer func(begin time.Time) {
pr, pw := io.Pipe()
defer pw.Close()
if err := r.ParseMultipartForm(32 << 20); err != nil {
http.Error(w, err.Error(), http.StatusBadRequest)
fmt.Printf("%v\n", err)
return
}
srcFile, handler, err := r.FormFile("video")
if err != nil {
http.Error(w, "INVALID_FILE", http.StatusBadRequest)
return
}
defer srcFile.Close()
// Copy uploaded file to *PipeWriter
go func() {
defer srcFile.Close()
defer pw.Close()
io.Copy(pw, srcFile)
}()
// Prep fileName
var srcFileName = strings.Replace(handler.Filename, " ", "_", -1)
var extension = filepath.Ext(srcFileName)
srcFileName = srcFileName[0 : len(srcFileName)-len(extension)]
// Create temp file to write too
f, err := ioutil.TempFile("./tmp/pending", srcFileName+"_")
if err != nil {
log.Fatal(err)
}
defer f.Close()
// Copy *PipeReader to temp file
io.Copy(f, pr)
contentType, err := getFileContentType("./" + f.Name())
if err != nil {
fmt.Printf("Content Type Error: %v\n", err)
return
}
fmt.Printf("ContentType = %v\n ", contentType)
if contentType != "video/x-flv" && contentType != "video/mp4" &&
contentType != "video/quicktime" && contentType != "video/x-msvideo" &&
contentType != "video/x-ms-wmv" && contentType != "video/webm" {
http.Error(w, "INVALID_FILE_TYPE", http.StatusBadRequest)
os.Remove(f.Name())
return
}
fmt.Printf("Upload path -> %v\n", f.Name())
upload := UploaderResponse{srcFileName, "./" + f.Name()}
resp, err := json.Marshal(upload)
if err != nil {
fmt.Printf("Error using Marshal %v\n", err)
}
w.Header().Set("Content-Type", "application/json")
w.WriteHeader(http.StatusOK)
status, err2 := w.Write(resp)
if err2 != nil {
fmt.Printf("Error when writing response to app %v\n", err2)
} else {
fmt.Printf("Status from uploading: %v\n", status)
}
_ = u.logger.Log("upload.Handler Duration ", time.Since(begin))
}(time.Now())
The code gets down to the last line, u.logger.Log()
which actually shows output. So the video is uploaded correctly...only responding to the client fails. Does anyone have ideas?
Disclaimer 2: I just found out that w.Write
returns an error, so I'm waiting on a file to finish uploading to see if I get an errors logged. Will respond with an update.
I uploaded a new file and the w.Write()
call works, returning a 91
integer. I'm not sure what the issue may be now. Could it be a time-out somewhere I don't know of. FYI, my server read and write timeouts are both 6 hours.
If it helps, here are the logs. It should be easy to follow the code with the outputs:
Upvotes: 0
Views: 746
Reputation: 1838
I was in the ballpark mentioning timeouts but it turns out this issue had nothing to do with the code. I stumbled upon another SO post that had the same problem but on AWS instead. The problem was a timeout from the load balancer. I used the instructions from Google Cloud's docs to increase the timeout from a default of 30 secs to a couple of hours. To be specific, the timeout setting is a count for how long to wait on the server to send a response to the client.
Upvotes: 0