CODEWITHSUNDEEP
kubernetes
mpen
mpen

Reputation: 283043

How to get access to filesystem with "kubectl debug" (ephemeral containers)?

If I do

POD=$($KUBECTL get pod -lsvc=app,env=production -o jsonpath="{.items[0].metadata.name}")
kubectl debug -it --image=mpen/tinker "$POD" -- zsh -i

I can get into a shell running inside my pod, but I want access to the filesystem for a container I've called "php". I think this should be at /proc/1/root/app but that directory doesn't exist. For reference, my Dockerfile has:

WORKDIR /app
COPY . .

So all the files should be in the root /app directory.

If I add --target=php then I get permission denied:

❯ cd /proc/1/root
cd: permission denied: /proc/1/root

How do I get access to the files?

Upvotes: 19

Views: 11730

Answers (2)

tlwhitec
tlwhitec

Reputation: 2473

The debug container has always its own filesystem, but - if the container runtime supports it - you should be able to use --target and access it via /proc/$PID/root just fine. One just needs to act as the exact same user (both $UID and $GID must match).

Example

First, prepare a container to be debugged:

kubectl run tested-pod --rm -it --image=busybox:latest --restart=Never \
  --overrides='
    {"spec": {
      "containers": [
        {
          "name":"tested-pod",
          "image":"busybox:latest",
          "stdin":true,
          "tty":true,
          "securityContext": {
            "runAsUser":1234,
            "runAsGroup":4321
          }
        }
      ]
    }
  }' \
  -- sh

This creates a new pod tested-pod where the container runs with UID=1234 and GID=4321, and gives you an interactive shell. Write something to the container's filesystem, so you can see it later:

touch /tmp/tested-pod-file
ls /tmp
# tested-pod-file

Next, open a new terminal and use the ephemeral container debug:

kubectl debug tested-pod -it --image=busybox:latest --target=tested-pod -- sh

At this moment, the debugging user is root, which can't access the FS, but we can already see the container's process:

id
# uid=0(root) gid=0(root) groups=0(root),10(wheel)
ls /proc/1/root
# ls: /proc/1/root: Permission denied
ps -o 'pid,user,group,comm'
# PID   USER     GROUP    COMMAND
#     1 1234     4321     sh
#     9 root     root     sh
#    25 root     root     ps

Now you gotta create a local group and user (name doesn't matter) and use it:

addgroup -g 4321 karel
adduser -HD -u 1234 -G karel karel
su karel

In the resulting shell you should be able to access the filesystem, even for writing (at least it works in my case):

ls /proc/1/root/tmp/
# tested-pod-file
touch /proc/1/root/tmp/debugger-file
ls /proc/1/root/tmp/
# debugger-file    tested-pod-file

You can verify this worked from the original container's shell:

ls /tmp
debugger-file    tested-pod-file

Upvotes: 29

larsks
larsks

Reputation: 311988

Reading through the documentation, using kubectl debug won't give you access to the filesystem in another container.

The simplest option may be to use kubectl exec to start a shell inside an existing container. There are some cases in which this isn't an option (for example, some containers contain only a single binary, and won't have a shell or other common utilities avaiable), but a php container will typically have a complete filesystem.

In this case, you can simply:

kubectl exec -it $POD -- sh

You can replace sh by bash or zsh depending on what shells are available in the existing image.

The linked documentation provides several other debugging options, but all involve working on copies of the pod.

Upvotes: 5

Related Questions