SSL3_GET_RECORD:wrong version number when loading to bigquery table from google cloud storage

Question

When loading to a Bigquery table from files in Google cloud storage, I kept getting this SSL3_GET_RECORD:wrong version number exception.

Eventually when I check the job history from the Google Bigquery job history webpage, the load job will show that it has succeeded.

Could you please help? Thank you.

Here is the error message I am getting:

========================================
== Platform ==
  CPython:2.7.6:Linux-2.6.18-194.32.1.el5-x86_64-with-redhat-5.5-Final
== bq version ==
  2.0.18
== Command line ==
  ['/opt/google-cloud-sdk/platform/bq/bq.py', '--credential_file', '/offworld/hornet/.config/gcloud/legacy_credentials/clok@vindicotech.com/singlestore.json', '--project', 'formal-cascade-571', 'load', '--source_format=NEWLINE_DELIMITED_JSON', 'dw_sandbox.impressions_20140603', 'gs://dw_sandbox/impressions/20140603/20140604175042285_20140604195938608_20140603_0_*', '/offworld/specificmedia/logsTobq/schemas/impressionsSchema.txt']
== UTC timestamp ==
  2014-06-05 01:19:06
== Error trace ==
  File "/opt/google-cloud-sdk/platform/bq/bq.py", line 779, in RunSafely
    return_value = self.RunWithArgs(*args, **kwds)
  File "/opt/google-cloud-sdk/platform/bq/bq.py", line 1020, in RunWithArgs
    job = client.Load(table_reference, source, schema=schema, **opts)
  File "/opt/google-cloud-sdk/platform/bq/bigquery_client.py", line 2011, in Load
    upload_file=upload_file, **kwds)
  File "/opt/google-cloud-sdk/platform/bq/bigquery_client.py", line 1611, in ExecuteJob
    job_id=job_id)
  File "/opt/google-cloud-sdk/platform/bq/bigquery_client.py", line 1599, in RunJobSynchronously
    result = self.WaitJob(job_reference)
  File "/opt/google-cloud-sdk/platform/bq/bigquery_client.py", line 1713, in WaitJob
    done, job = self.PollJob(job_reference, status=status, wait=wait)
  File "/opt/google-cloud-sdk/platform/bq/bigquery_client.py", line 1752, in PollJob
    job = self.apiclient.jobs().get(**dict(job_reference)).execute()
  File "/opt/google-cloud-sdk/platform/bq/bigquery_client.py", line 307, in execute
    return super(BigqueryHttp, self).execute(**kwds)
  File "/opt/google-cloud-sdk/bin/bootstrapping/../../lib/oauth2client/util.py", line 132, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "/opt/google-cloud-sdk/bin/bootstrapping/../../lib/apiclient/http.py", line 716, in execute
    body=self.body, headers=self.headers)
  File "/opt/google-cloud-sdk/bin/bootstrapping/../../lib/oauth2client/util.py", line 132, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "/opt/google-cloud-sdk/bin/bootstrapping/../../lib/oauth2client/client.py", line 490, in new_request
    redirections, connection_type)
  File "/opt/google-cloud-sdk/bin/bootstrapping/../../lib/httplib2/__init__.py", line 1586, in request
    (response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey)
  File "/opt/google-cloud-sdk/bin/bootstrapping/../../lib/httplib2/__init__.py", line 1333, in _request
    (response, content) = self._conn_request(conn, request_uri, method, body, headers)
  File "/opt/google-cloud-sdk/bin/bootstrapping/../../lib/httplib2/__init__.py", line 1289, in _conn_request
    response = conn.getresponse()
  File "/usr/lib64/python2.7/httplib.py", line 1045, in getresponse
    response.begin()
  File "/usr/lib64/python2.7/httplib.py", line 409, in begin
    version, status, reason = self._read_status()
  File "/usr/lib64/python2.7/httplib.py", line 365, in _read_status
    line = self.fp.readline(_MAXLINE + 1)
  File "/usr/lib64/python2.7/socket.py", line 476, in readline
    data = self._sock.recv(self._rbufsize)
  File "/usr/lib64/python2.7/ssl.py", line 241, in recv
    return self.read(buflen)
  File "/usr/lib64/python2.7/ssl.py", line 160, in read
    return self._sslobj.read(len)
========================================

Unexpected exception in load operation: [Errno 1] _ssl.c:1426:
error:1408F10B:SSL routines:SSL3_GET_RECORD:wrong version number

Answer 1

Is there any chance that you're using the same HTTP object in multiple threads? I.e the thread you create the job in is not necessarily the same thread you poll for completion in? If this is the case, this came up internally within google today, and this was the fix:

class _HTTPFactoryWrapper(object):
  """Wraps a request factory so that each request returns a new http object.

  API client's Http object is not threadsafe since calls to the same domain will
  reuse the same HTTPConnection.  If one API call is outstanding then a second
  will try to send a request over the same domain.  This causes chaos that
  seems to surface itself as SSLErrors during processing.

  """

  def __init__(self, factory):
    self.factory = factory

  def request(self, *args, **kwargs):
    return self.factory.Create().request(*args, **kwargs)

Then change the creation of the bigquery stub from:

 return discovery.build(api_name, api_version, http=http_factory.Create())

To

  http_wrapper = _HTTPFactoryWrapper(http_factory)
  return discovery.build(api_name, api_version, http=http_wrapper)