CODEWITHSUNDEEP
google-app-enginecurl
bjoseru
bjoseru

Reputation: 170

Google cloud storage upload using generic signed url and curl

I am trying to put a file into a Google Cloud Storage (GCS) bucket from the command line. At a later stage this shall be used in a deployed script at the user end without any type of user-visible authentication.

So far I generate a signed url like this:

gsutil signurl -p notasecret -m PUT -d 1d myserviceaccount.p12 gs://mybucket/testfile

which will generate something like

https://storage.googleapis.com/mybucket/testfile?GoogleAccessId=myserviceaccount@developer.gserviceaccount.com&Expires=1430963040&Signature=gMf2h95bNmolizUGYrsQ%2F%2F%2FiHxW14I%2F0EOU3ZSFWtfCwNqSyok3iweQiuPxYXH4b26FeDSrmFOXB58%2B%2B%2BiAOJ%2B1gdLC9Y%2BkeUdbrjH0eGTW0NVsM1AWY2LsQ3dYf5Ho%2Bos1Fk26EsLJlD096Ku9aWqLW%2FpL%2FBSsUIfHijrFJPdI%3D

The next step (at the user end) would be curl uploading the file with a PUT request. Like so:

curl -X PUT --data-binary @testfile 'https://storage.googleapis.com/mybucket/testfile?GoogleAccessId=myserviceaccount@developer.gserviceaccount.com&Expires=1430963040&Signature=gMf2h95bNmolizUGYrsQ%2F%2F%2FiHxW14I%2F0EOU3ZSFWtfCwNqSyok3iweQiuPxYXH4b26FeDSrmFOXB58%2B%2B%2BiAOJ%2B1gdLC9Y%2BkeUdbrjH0eGTW0NVsM1AWY2LsQ3dYf5Ho%2Bos1Fk26EsLJlD096Ku9aWqLW%2FpL%2FBSsUIfHijrFJPdI%3D'

I can get this to work with an existing file in the bucket and a GET request (for downloading), but it does not seem to work for uploading. curl throws the server's response with error messages like this at me:

<?xml version='1.0' encoding='UTF-8'?>
<Error>
<Code>SignatureDoesNotMatch</Code>
<Message>The request signature we calculated does not match the signature you provided.
         Check your Google secret key and signing method.</Message>
<StringToSign>PUT

application/x-www-form-urlencoded
1430963040
/mybucket/testfile</StringToSign>
</Error>

And this makes sense to me, as obviously I am not just making a bare PUT request, but one for a particular file of a specific size, whereas the signature computed by 'gsutil signurl' would not know about these details at the time it is computed.

Somehow I was under the impression (e.g., based on the last usage case described in gsutil signurl documentation and also in the post How to allow anonymous uploads to cloud storage) that it should be possible to generate a generic signed url for uploading purposes and then use it later. Am I just mistaken about this point or is there a way to fix the curl request?

Any thoughts about this are appreciated. However, I'd like this to work with "minimal tools", i.e., ideally shell and curl only, but no other programming languages.

EDIT: Organising one's thoughts by formulating the exact problem is the first step towards the solution. I realise now that 

curl -X PUT -T - [request-url] < testfile

does actually solve the immediate problem. However, this means multiple users would write to the same file if they use the same signed url. The documentation suggests you can omit the object name in the creation of the signed url, i.e., use 

gsutil signurl -p notasecret -m PUT -d 1d myserviceaccount.p12 gs://mybucket/

This, supposedly, would allow anyone with the resulting signed url to put any object of any type into my bucket. Only I do not get this work, as I don't see how you can then tell GCS which object you are actually writing to.

Upvotes: 16

Views: 8462

Answers (3)

Oliver Shaw
Oliver Shaw

Reputation: 5422

This was driving me mad too. Turns out it was the binary-file part of the curl command. Try this instead:

curl -X PUT --upload-file me.jpeg $SIGNED_URL

Upvotes: 6

shockwave
shockwave

Reputation: 145

I encountered the similar problem(403 forbidden).
It turned out that my json library, which I use it to marshal each response, would replace & by \u0026 for security concern. So the url may be correct in the program but invalid in client side
So I guess that there might be some string encoding bug inside the Signature query string of your url since the signature string is harder to detect error in comparison with my \u0026.

Upvotes: 0

Jose L Ugia
Jose L Ugia

Reputation: 6250

If the resource does not specify a single object, you can do so on an individual basis by adding a URL param to the request with the name of the object. For example:

curl -X PUT -T - [request-url]?name=[object-name] < testfile

This surely works with storage/v1, although I have not tried myself with a signed URL yet.

Upvotes: 1

Related Questions