CODEWITHSUNDEEP
gokuberneteskubernetes-helmkongkong-plugin
Chalukya J
Chalukya J

Reputation: 51

Kong custom golang plugin not working in kubernetes/helm setup

I have written custom golang kong plugin called go-wait following the example from the github repo https://github.com/redhwannacef/youtube-tutorials/tree/main/kong-gateway-custom-plugin

The only difference is I created a custom docker image so kong would have the mentioned plugin by default in it's /usr/local/bin directory Here's the dockerfile

FROM golang:1.18.3-alpine as pluginbuild
COPY ./charts/custom-plugins/ /app/custom-plugins
RUN cd /app/custom-plugins && \
    for d in ./*/ ; do (cd "$d" && go mod tidy && GOOS=linux GOARCH=amd64 go build .); done

RUN mkdir /app/all-plugin-execs && cd /app/custom-plugins && \
    find . -type f -not -name "*.*" | xargs -i cp {} /app/all-plugin-execs/

FROM kong:2.8

COPY --from=pluginbuild /app/all-plugin-execs/ /usr/local/bin/
COPY --from=pluginbuild /app/all-plugin-execs/ /usr/local/bin/plugin-ref/

# Loop through the plugin-ref directory and create an entry for all of them in 
# both KONG_PLUGINS and KONG_PLUGINSERVER_NAMES env vars respectively
# Additionally append `bundled` to KONG_PLUGINS list as without it any unused plugin will case Kong to error out

#### Example Env vars for a plugin named `go-wait`
# ENV KONG_PLUGINS=go-wait
# ENV KONG_PLUGINSERVER_NAMES=go-wait
# ENV KONG_PLUGINSERVER_GO_WAIT_QUERY_CMD="/usr/local/bin/go-wait -dump"
####
RUN cd /usr/local/bin/plugin-ref/ && \
    PLUGINS=$(ls | tr '\n' ',') && PLUGINS=${PLUGINS::-1} && \
    echo -e "KONG_PLUGINS=bundled,$PLUGINS\nKONG_PLUGINSERVER_NAMES=$PLUGINS" >> ~/.bashrc

# Loop through the plugin-ref directory and create an entry for QUERY_CMD entries needed to load the plugin
# format KONG_PLUGINSERVER_EG_PLUGIN_QUERY_CMD if the plugin name is `eg-plugin` and it should point to the 
# plugin followed by `-dump` argument
RUN cd /usr/local/bin/plugin-ref/ && \
    for f in *; do echo "$f" | tr "[:lower:]" "[:upper:]" | tr '-' '_' | \
    xargs -I {} sh -c "echo 'KONG_PLUGINSERVER_{}_QUERY_CMD=' && echo '\"/usr/local/bin/{} -dump\"' | tr [:upper:] [:lower:] | tr '_' '-'" | \
    sed -e '$!N;s/\n//' | xargs -i echo "{}" >> ~/.bashrc; done

This works fine in the docker-compose file and docker container. But when I tried to use the same image in the kubernetes environment along with kong-ingress-controller, I started running into errors "failed to fill-in defaults for plugin: go-wait" and/or a bunch of other errors including "plugin 'go-wait' enabled but not installed" in the logs and I ended up not being able to enable it.

Has anyone tried including go plugins in their kubernetes/helm kong setup. If so please shed some light on this

Upvotes: 3

Views: 1120

Answers (1)

Chalukya J
Chalukya J

Reputation: 51

Update: Found the answer I was looking for, along with setting the environment variables generated by the image, there's modifications in the _helpers.tpl file of the kong helm chart itself. The reason is that in the deployment charts, the configuration expects plugins to be configured in values-custom.yml used to override the default settings.

But the helm chart seems to be specific to values and plugins being loaded via configMaps which turned out to be a huge bottleneck, as any binary plugin you will generate in golang for kong is going to exceed the maximum allowed limit of the configMaps in kubernetes. That's the whole reason I had set out on this endeavor to make the plugins part of my image.

TL;dr I was able to clone the repo to my local system, make the changes for the following patch for loading the plugins from values without having to club them with the lua plugins. (Credits : Answer of thatbenguy from the discussion https://discuss.konghq.com/t/how-to-load-go-plugins-using-kong-helm-chart/5717/10)

--- a/charts/kong/templates/_helpers.tpl
+++ b/charts/kong/templates/_helpers.tpl
@@ -530,6 +530,9 @@ The name of the service used for the ingress controller's validation webhook

 {{- define "kong.plugins" -}}
 {{ $myList := list "bundled" }}
+{{- range .Values.plugins.goPlugins -}}
+{{- $myList = append $myList .pluginName -}}
+{{- end -}}
 {{- range .Values.plugins.configMaps -}}
 {{- $myList = append $myList .pluginName -}}
 {{- end -}}

Add the following block to my values-custom.yml and I was good to go.

Hopefully this helps anyone else also trying to write custom plugins for kong in golang for use in helm charts.

env:
  database: "off"
  plugins: bundled,go-wait
  pluginserver_names: go-wait
  pluginserver_go_wait_query_cmd: "/usr/local/bin/go-wait -dump"
  

plugins:
  goPlugins:
    - pluginName: "go-wait"

NOTE : Please remember all this still depends on you having the prebuilt custom kong plugins in your image, in my case I had built an image from the above dockerfile contents (in question) and pushed that to my own docker hub repo and replaced the image in the values-custom.yml using the following block

image:
  repository: chalukyaj/kong-custom-image
  tag: "1.0.1"

PS: As you guys might have noticed, the only disappointment I have with this is that the environment variables couldn't just be picked from the docker image's ~/.bashrc, which would have made this awesome. But nonetheless, this works, and I couldn't find a single post which showed how to use the new go-pdk (instead of the older go-pluginserver) to build the go plugins and use them in helm.

Upvotes: 2

Related Questions