Transfer from AWS S3 to Google-Cloud-Storage "UNKNOWN" failures

Question

I am using the Transfer tool provided in the GCS web dashboard/console to schedule a copy of backup files from S3 to GCS.

In the GCS transfer logs it shows "UNKNOWN: (showing 5 of 241 failures)". There is nothing different about the files shown in that failure list. The same 241 files are failing at each scheduled transfer. Is there somewhere I can get an actual error message?

I have already checked:

• There are over 1.5 million other files successfully copied before and after these failures. i.e. These failures are not at the beginning or end of the list of transfers, nor are they sequential.
• The file sizes are not too large. Size of failed transfers range from 100KB to 7MB.
• File names have allowed special characters e.g. _ - spaces, these characters are present in failed transfers and successful transfers.
• Failed files have exact same permissions as successfully transferred files.
• The policy for the IAM user used in the transfer grants all actions to S3 bucket.
• Confirmed IAM policy using same access token and secret with s3cmd.
• Checked S3 access logs for further info/errors ... shows http 206 status for both successful transfers and failed transfers. ( Not the http status I would have expected but the same appears for successful and failed transfers)

Answer 1

There are a few things that can cause UNKNOWN errors. Some are transient issues in the service (in which case, you should re-run your transfer--only new/changed objects will be copied the second time).

In other cases, it can be quirks in your source objects. For example, the service will reject objects that do not have a Content-Type that conforms to RFC 1341. In particular, it needs to match the pattern /, with additional restrictions. See What are all the possible values for HTTP "Content-Type" header?