Running k8s_exec from Ansible playbook results in Handshake status 403 Forbidden error

Question

I need to be able to execute some shell scripts inside my container from Ansible playbook. Ansible play is running in another container in my cluster. Cluster is in RedHat OpenShift 4.3.

I am using the k8s_exec from community.kubernetes to do this:

- name: k8s_exec Execute {{ script }} {{ script_args }} command in pod {{ the_pod }} 
  k8s_exec:
    namespace: "{{ meta.name }}"
    pod: "{{ the_pod }}"
    command: "{{ script }} {{ script_args }}" 
  register: call_result
  when: the_pod is defined

- name: The result of running {{ script }} {{ script_args }} in {{ the_pod }} 
  debug:
    var: call_result

But I get the following:

--------------------------- Ansible Task StdOut -------------------------------

 TASK [The result of running /elm/server/containerState.sh setup in ibmjazz-jts-854b4dd7bc-ptdgw] ******************************** 
 [0;32mok: [localhost] => { [0m
 [0;32m    "call_result": { [0m
 [0;32m        "changed": false, [0m
 [0;32m        "exception": "Traceback (most recent call last):\n  File \"/usr/local/lib/python3.6/site-packages/kubernetes/stream/ws_client.py\", line 254, in websocket_call\n    client = WSClient(configuration, get_websocket_url(url), headers)\n  File \"/usr/local/lib/python3.6/site-packages/kubernetes/stream/ws_client.py\", line 77, in __init__\n    self.sock.connect(url, header=header)\n  File \"/usr/local/lib/python3.6/site-packages/websocket/_core.py\", line 226, in connect\n    self.handshake_response = handshake(self.sock, *addrs, **options)\n  File \"/usr/local/lib/python3.6/site-packages/websocket/_handshake.py\", line 80, in handshake\n    status, resp = _get_resp_headers(sock)\n  File \"/usr/local/lib/python3.6/site-packages/websocket/_handshake.py\", line 165, in _get_resp_headers\n    raise WebSocketBadStatusException(\"Handshake status %d %s\", status, status_message, resp_headers)\nwebsocket._exceptions.WebSocketBadStatusException: Handshake status 403 Forbidden\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n  File \"/opt/ansible/.ansible/tmp/ansible-tmp-1591037058.988166-154830146809790/AnsiballZ_k8s_exec.py\", line 102, in <module>\n    _ansiballz_main()\n  File \"/opt/ansible/.ansible/tmp/ansible-tmp-1591037058.988166-154830146809790/AnsiballZ_k8s_exec.py\", line 94, in _ansiballz_main\n    invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)\n  File \"/opt/ansible/.ansible/tmp/ansible-tmp-1591037058.988166-154830146809790/AnsiballZ_k8s_exec.py\", line 40, in invoke_module\n    runpy.run_module(mod_name='ansible_collections.community.kubernetes.plugins.modules.k8s_exec', init_globals=None, run_name='__main__', alter_sys=True)\n  File \"/usr/lib64/python3.6/runpy.py\", line 205, in run_module\n    return _run_module_code(code, init_globals, run_name, mod_spec)\n  File \"/usr/lib64/python3.6/runpy.py\", line 96, in _run_module_code\n    mod_name, mod_spec, pkg_name, script_name)\n  File \"/usr/lib64/python3.6/runpy.py\", line 85, in _run_code\n    exec(code, run_globals)\n  File \"/tmp/ansible_k8s_exec_payload_0wco0o8m/ansible_k8s_exec_payload.zip/ansible_collections/community/kubernetes/plugins/modules/k8s_exec.py\", line 148, in <module>\n  File \"/tmp/ansible_k8s_exec_payload_0wco0o8m/ansible_k8s_exec_payload.zip/ansible_collections/community/kubernetes/plugins/modules/k8s_exec.py\", line 135, in main\n  File \"/usr/local/lib/python3.6/site-packages/kubernetes/stream/stream.py\", line 36, in stream\n    return func(*args, **kwargs)\n  File \"/usr/local/lib/python3.6/site-packages/kubernetes/client/apis/core_v1_api.py\", line 835, in connect_get_namespaced_pod_exec\n    (data) = self.connect_get_namespaced_pod_exec_with_http_info(name, namespace, **kwargs)\n  File \"/usr/local/lib/python3.6/site-packages/kubernetes/client/apis/core_v1_api.py\", line 935, in connect_get_namespaced_pod_exec_with_http_info\n    collection_formats=collection_formats)\n  File \"/usr/local/lib/python3.6/site-packages/kubernetes/client/api_client.py\", line 344, in call_api\n    _return_http_data_only, collection_formats, _preload_content, _request_timeout)\n  File \"/usr/local/lib/python3.6/site-packages/kubernetes/client/api_client.py\", line 178, in __call_api\n    _request_timeout=_request_timeout)\n  File \"/usr/local/lib/python3.6/site-packages/kubernetes/stream/stream.py\", line 31, in _intercept_request_call\n    return ws_client.websocket_call(config, *args, **kwargs)\n  File \"/usr/local/lib/python3.6/site-packages/kubernetes/stream/ws_client.py\", line 260, in websocket_call\n    raise ApiException(status=0, reason=str(e))\nkubernetes.client.rest.ApiException: (0)\nReason: Handshake status 403 Forbidden\n\n", [0m
 [0;32m        "failed": true, [0m
 [0;32m        "module_stderr": "Traceback (most recent call last):\n  File \"/usr/local/lib/python3.6/site-packages/kubernetes/stream/ws_client.py\", line 254, in websocket_call\n    client = WSClient(configuration, get_websocket_url(url), headers)\n  File \"/usr/local/lib/python3.6/site-packages/kubernetes/stream/ws_client.py\", line 77, in __init__\n    self.sock.connect(url, header=header)\n  File \"/usr/local/lib/python3.6/site-packages/websocket/_core.py\", line 226, in connect\n    self.handshake_response = handshake(self.sock, *addrs, **options)\n  File \"/usr/local/lib/python3.6/site-packages/websocket/_handshake.py\", line 80, in handshake\n    status, resp = _get_resp_headers(sock)\n  File \"/usr/local/lib/python3.6/site-packages/websocket/_handshake.py\", line 165, in _get_resp_headers\n    raise WebSocketBadStatusException(\"Handshake status %d %s\", status, status_message, resp_headers)\nwebsocket._exceptions.WebSocketBadStatusException: Handshake status 403 Forbidden\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n  File \"/opt/ansible/.ansible/tmp/ansible-tmp-1591037058.988166-154830146809790/AnsiballZ_k8s_exec.py\", line 102, in <module>\n    _ansiballz_main()\n  File \"/opt/ansible/.ansible/tmp/ansible-tmp-1591037058.988166-154830146809790/AnsiballZ_k8s_exec.py\", line 94, in _ansiballz_main\n    invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)\n  File \"/opt/ansible/.ansible/tmp/ansible-tmp-1591037058.988166-154830146809790/AnsiballZ_k8s_exec.py\", line 40, in invoke_module\n    runpy.run_module(mod_name='ansible_collections.community.kubernetes.plugins.modules.k8s_exec', init_globals=None, run_name='__main__', alter_sys=True)\n  File \"/usr/lib64/python3.6/runpy.py\", line 205, in run_module\n    return _run_module_code(code, init_globals, run_name, mod_spec)\n  File \"/usr/lib64/python3.6/runpy.py\", line 96, in _run_module_code\n    mod_name, mod_spec, pkg_name, script_name)\n  File \"/usr/lib64/python3.6/runpy.py\", line 85, in _run_code\n    exec(code, run_globals)\n  File \"/tmp/ansible_k8s_exec_payload_0wco0o8m/ansible_k8s_exec_payload.zip/ansible_collections/community/kubernetes/plugins/modules/k8s_exec.py\", line 148, in <module>\n  File \"/tmp/ansible_k8s_exec_payload_0wco0o8m/ansible_k8s_exec_payload.zip/ansible_collections/community/kubernetes/plugins/modules/k8s_exec.py\", line 135, in main\n  File \"/usr/local/lib/python3.6/site-packages/kubernetes/stream/stream.py\", line 36, in stream\n    return func(*args, **kwargs)\n  File \"/usr/local/lib/python3.6/site-packages/kubernetes/client/apis/core_v1_api.py\", line 835, in connect_get_namespaced_pod_exec\n    (data) = self.connect_get_namespaced_pod_exec_with_http_info(name, namespace, **kwargs)\n  File \"/usr/local/lib/python3.6/site-packages/kubernetes/client/apis/core_v1_api.py\", line 935, in connect_get_namespaced_pod_exec_with_http_info\n    collection_formats=collection_formats)\n  File \"/usr/local/lib/python3.6/site-packages/kubernetes/client/api_client.py\", line 344, in call_api\n    _return_http_data_only, collection_formats, _preload_content, _request_timeout)\n  File \"/usr/local/lib/python3.6/site-packages/kubernetes/client/api_client.py\", line 178, in __call_api\n    _request_timeout=_request_timeout)\n  File \"/usr/local/lib/python3.6/site-packages/kubernetes/stream/stream.py\", line 31, in _intercept_request_call\n    return ws_client.websocket_call(config, *args, **kwargs)\n  File \"/usr/local/lib/python3.6/site-packages/kubernetes/stream/ws_client.py\", line 260, in websocket_call\n    raise ApiException(status=0, reason=str(e))\nkubernetes.client.rest.ApiException: (0)\nReason: Handshake status 403 Forbidden\n\n", [0m
 [0;32m        "module_stdout": "", [0m
 [0;32m        "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error", [0m
 [0;32m        "rc": 1 [0m
 [0;32m    } [0m
 [0;32m} [0m
 [0;32m [

The role has all the correct permissions (verbs: create, get) for pods/exec. I added second explicit apiGroups block of permissions; originally it was only the first one. Does not make any difference though.

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  creationTimestamp: null
  name: my-operator
rules:
- apiGroups:
  - "*"
  resources:
  - "*"
  verbs:
  - '*'
- apiGroups:
  - ""
  resources:
  - pods
  - pods/exec
  - pods/log
  verbs:
  - create
  - delete
  - get
  - list
  - patch
  - update
  - watch

What am I missing?

Thanks in advance

Alex

Answer 1

For anyone who ends up here using the sdk-operator, MAKE SURE YOU HAVE - pods/exec In your resources for the role.

Answer 2

I have discovered the problem, there was a typo in the value of {{ meta.name }}.