CODEWITHSUNDEEP
javagoogle-cloud-storagepre-signed-url
Jaiganesh Prabhakaran
Jaiganesh Prabhakaran

Reputation: 1

Generate gcs signed URL to upload large files

I have been trying to upload large files by generating a signed URL. Here is the documentation I was following to generate the signed URL: https://cloud.google.com/storage/docs/access-control/signing-urls-with-helpers#code-samples

It was working fine for files in 100's of MB's but as soon as the file size went up to 1 GB, the curl command started timing out even after increasing the expiration time. I tried looking at the answer here: https://stackoverflow.com/a/63789297/7466551, but I am still unable to get the URL working to upload the URL.

I am using this command to upload the file: curl -X POST -H 'x-goog-resumable: start' --upload-file file-name 'pre_signed_google_url'. I am adding the 'x-goog-resumable: start' header as I am having "x-goog-resumable", "start" header as a part of my code to generate the URL.

Can someone please let me know if I need to do any additional thing to generate the URL to upload large files?

Upvotes: 0

Views: 1636

Answers (1)

Jaiganesh Prabhakaran
Jaiganesh Prabhakaran

Reputation: 1

Answering my own question as I had to use two separate sources to reach a solution.

On top of the java code here: https://stackoverflow.com/a/63789297/7466551, I referred to the medium article here: https://medium.com/google-cloud/google-cloud-storage-signedurl-resumable-upload-with-curl-74f99e41f0a2

So you need the following additional lines of code on top of the StackOverflow answer to get a signed URL for resumable uploads:

// Open a HTTP connection and add header
URL obj = new URL(url.toString());
HttpURLConnection connection = (HttpURLConnection) obj.openConnection();
connection.setRequestMethod("POST");
connection.setRequestProperty("x-goog-resumable", "start");
connection.setDoOutput(true);

// Connect to the URL and post headers
DataOutputStream writer = new DataOutputStream(
        connection.getOutputStream());
writer.writeBytes("");
writer.flush();
writer.close();

// Checking the responseCode to
if (connection.getResponseCode() == connection.HTTP_CREATED) {
    connection.disconnect();
    System.out.println("Location: " + connection.getHeaderField("Location"));
}
else {
    // Throw error
    BufferedReader reader = new BufferedReader(new InputStreamReader(connection.getInputStream()));
    String inputLine;
    StringBuffer response = new StringBuffer();
    while ((inputLine = reader.readLine()) != null) {
        response.append(inputLine);
    }
    reader.close();
    String errorMessage = response.toString();
    connection.disconnect();
    throw new IOException(errorMessage);

}

Upvotes: 0

Related Questions