CODEWITHSUNDEEP
kubernetesworkflowminikubeminioargo-workflows
Neeraj Sirdeshmukh
Neeraj Sirdeshmukh

Reputation: 11

Error in Argo workflow run on local Minikube K8s cluster with MinIO as the artifact repository

I'm running an Argo workflow on a local MinIO K8s cluster. I'm setting up an Artifact Repository on MinIO where output artifacts from my workflow can be stored. I followed the instructions here https://argoproj.github.io/argo-workflows/configure-artifact-repository/#configuring-minio .

The error I'm running into is: failed to create new S3 client: Endpoint url cannot have fully qualified paths.

My MinIO endpoint is at http://127.0.0.1:52139.

Here is my workflow YAML file:

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: artifactory-repository-ref-
spec:
  archiveLogs: true
  entrypoint: main
  templates:
    - name: main
      container:
        image: docker/whalesay:latest
        command: [ sh, -c ]
        args: [ "cowsay hello world | tee /tmp/hello_world.txt" ]
      archiveLocation:
        archiveLogs: true
      outputs:
        artifacts:
          - name: hello_world
            path: /tmp/hello_world.txt

Here is my workflow-controller-configmap YAML which is deployed in the same namespace as the workflow:

# This file describes the config settings available in the workflow controller configmap
apiVersion: v1
kind: ConfigMap
metadata:
  name: workflow-controller-configmap
data:                      # "config: |" key is optional in 2.7+!
  artifactRepository: |    # However, all nested maps must be strings
   archiveLogs: true
   s3:
     endpoint: argo-artifacts:9000
     bucket: my-bucket
     insecure: true
     accessKeySecret:                #omit if accessing via AWS IAM
       name: my-minio-cred
       key: accessKey
     secretKeySecret:                #omit if accessing via AWS IAM
       name: my-minio-cred
       key: secretKey
     useSDKCreds: true

I've created a secret called my-minio-cred in the same namespace where the workflow is running.

Here are the logs from the pod where the workflow is running:

time="2023-02-16T21:39:05.044Z" level=info msg="Starting Workflow Executor" version=v3.4.5
time="2023-02-16T21:39:05.047Z" level=info msg="Using executor retry strategy" Duration=1s Factor=1.6 Jitter=0.5 Steps=5
time="2023-02-16T21:39:05.047Z" level=info msg="Executor initialized" deadline="0001-01-01 00:00:00 +0000 UTC" includeScriptOutput=false namespace=argo podName=artifactory-repository-ref-5tcmt template="{\"name\":\"main\",\"inputs\":{},\"outputs\":{\"artifacts\":[{\"name\":\"hello_world\",\"path\":\"/tmp/hello_world.txt\"}]},\"metadata\":{},\"container\":{\"name\":\"\",\"image\":\"docker/whalesay:latest\",\"command\":[\"sh\",\"-c\"],\"args\":[\"cowsay hello world | tee /tmp/hello_world.txt\"],\"resources\":{}},\"archiveLocation\":{\"archiveLogs\":true,\"s3\":{\"endpoint\":\"http://127.0.0.1:52897\",\"bucket\":\"my-bucket\",\"insecure\":true,\"accessKeySecret\":{\"name\":\"my-minio-cred\",\"key\":\"accessKey\"},\"secretKeySecret\":{\"name\":\"my-minio-cred\",\"key\":\"secretKey\"},\"useSDKCreds\":true,\"key\":\"artifactory-repository-ref-5tcmt/artifactory-repository-ref-5tcmt\"}}}" version="&Version{Version:v3.4.5,BuildDate:2023-02-07T12:36:25Z,GitCommit:1253f443baa8ad1610d2e62ec26ecdc85fe1b837,GitTag:v3.4.5,GitTreeState:clean,GoVersion:go1.18.10,Compiler:gc,Platform:linux/arm64,}"
time="2023-02-16T21:39:05.047Z" level=info msg="Starting deadline monitor"
time="2023-02-16T21:39:08.048Z" level=info msg="Main container completed" error="<nil>"
time="2023-02-16T21:39:08.048Z" level=info msg="No Script output reference in workflow. Capturing script output ignored"
time="2023-02-16T21:39:08.048Z" level=info msg="No output parameters"
time="2023-02-16T21:39:08.048Z" level=info msg="Saving output artifacts"
time="2023-02-16T21:39:08.048Z" level=info msg="stopping progress monitor (context done)" error="context canceled"
time="2023-02-16T21:39:08.048Z" level=info msg="Deadline monitor stopped"
time="2023-02-16T21:39:08.048Z" level=info msg="Staging artifact: hello_world"
time="2023-02-16T21:39:08.049Z" level=info msg="Copying /tmp/hello_world.txt from container base image layer to /tmp/argo/outputs/artifacts/hello_world.tgz"
time="2023-02-16T21:39:08.049Z" level=info msg="/var/run/argo/outputs/artifacts/tmp/hello_world.txt.tgz -> /tmp/argo/outputs/artifacts/hello_world.tgz"
time="2023-02-16T21:39:08.049Z" level=info msg="S3 Save path: /tmp/argo/outputs/artifacts/hello_world.tgz, key: artifactory-repository-ref-5tcmt/artifactory-repository-ref-5tcmt/hello_world.tgz"
time="2023-02-16T21:39:08.049Z" level=info msg="Creating minio client using static credentials" endpoint="http://127.0.0.1:52897"
time="2023-02-16T21:39:08.049Z" level=warning msg="Non-transient error: Endpoint url cannot have fully qualified paths."
time="2023-02-16T21:39:08.049Z" level=info msg="Save artifact" artifactName=hello_world duration="282.917µs" error="failed to create new S3 client: Endpoint url cannot have fully qualified paths." key=artifactory-repository-ref-5tcmt/artifactory-repository-ref-5tcmt/hello_world.tgz
time="2023-02-16T21:39:08.049Z" level=error msg="executor error: failed to create new S3 client: Endpoint url cannot have fully qualified paths."
time="2023-02-16T21:39:08.049Z" level=info msg="S3 Save path: /tmp/argo/outputs/logs/main.log, key: artifactory-repository-ref-5tcmt/artifactory-repository-ref-5tcmt/main.log"
time="2023-02-16T21:39:08.049Z" level=info msg="Creating minio client using static credentials" endpoint="http://127.0.0.1:52897"
time="2023-02-16T21:39:08.049Z" level=warning msg="Non-transient error: Endpoint url cannot have fully qualified paths."
time="2023-02-16T21:39:08.049Z" level=info msg="Save artifact" artifactName=main-logs duration="28.5µs" error="failed to create new S3 client: Endpoint url cannot have fully qualified paths." key=artifactory-repository-ref-5tcmt/artifactory-repository-ref-5tcmt/main.log
time="2023-02-16T21:39:08.049Z" level=error msg="executor error: failed to create new S3 client: Endpoint url cannot have fully qualified paths."
time="2023-02-16T21:39:08.056Z" level=info msg="Create workflowtaskresults 403"
time="2023-02-16T21:39:08.056Z" level=warning msg="failed to patch task set, falling back to legacy/insecure pod patch, see https://argoproj.github.io/argo-workflows/workflow-rbac/" error="workflowtaskresults.argoproj.io is forbidden: User \"system:serviceaccount:argo:default\" cannot create resource \"workflowtaskresults\" in API group \"argoproj.io\" in the namespace \"argo\""
time="2023-02-16T21:39:08.057Z" level=info msg="Patch pods 403"
time="2023-02-16T21:39:08.057Z" level=warning msg="Non-transient error: pods \"artifactory-repository-ref-5tcmt\" is forbidden: User \"system:serviceaccount:argo:default\" cannot patch resource \"pods\" in API group \"\" in the namespace \"argo\""
time="2023-02-16T21:39:08.057Z" level=error msg="executor error: pods \"artifactory-repository-ref-5tcmt\" is forbidden: User \"system:serviceaccount:argo:default\" cannot patch resource \"pods\" in API group \"\" in the namespace \"argo\""
time="2023-02-16T21:39:08.057Z" level=info msg="Alloc=6350 TotalAlloc=12366 Sys=18642 NumGC=4 Goroutines=5"
time="2023-02-16T21:39:08.057Z" level=fatal msg="failed to create new S3 client: Endpoint url cannot have fully qualified paths."

I've tried changing the endpoint key in the workflow-controller-config.yaml from 127.0.0.1:52139 to 127.0.0.1:9000 and also argo-artifacts:9000 but it still doesn't work. argo-artifacts is the name of the LoadBalancer service thats created by the helm install argo-artifacts minio/minio command.

I got the endpoint of the MinIO bucket from minikube service --url argo-artifacts as given in the 'Configuring MinIO' section at https://argoproj.github.io/argo-workflows/configure-artifact-repository/#configuring-minio

Everything is in the same namespace.

What could be wrong here?

I tried changing the endpoint URL of the MinIO bucket, changing namespaces for different components, and changing the namespace that the argo-artifacts service gets deployed in.

Upvotes: 1

Views: 2661

Answers (2)

che---
che---

Reputation: 69

May you check this issue, IMHO it is related to your problem: quick start fails out of the box due to RBAC error

You can try to workaround this by

  • using the option --serviceaccount argo or
  • deploying Argo Workflows by using the installer for the latest getting started guide: Quick Start, applying the manifest quick-start-minimal.yaml

Upvotes: 1

CorneliuC
CorneliuC

Reputation: 1

There are multiple problems in your log. One of them is:

time="2023-02-16T21:39:08.049Z" level=info msg="Creating minio client using static credentials" endpoint="http://127.0.0.1:52897"
time="2023-02-16T21:39:08.049Z" level=warning msg="Non-transient error: Endpoint url cannot have fully qualified paths."

First of all - I don't think your minio endpoint will be exposed on a localhost (127.0.0.1) IP address, not inside the minikube cluster anyway. I am assuming this value is being picked up from the "my-minio-cred" secret, so please change it there to "argo-artifacts:9000". Second - please note that you don't need to have a "http" there, this is what the above warning is about.

Another problem that I am seeing -

time="2023-02-16T21:39:08.056Z" level=warning msg="failed to patch task set, falling back to legacy/insecure pod patch, see https://argoproj.github.io/argo-workflows/workflow-rbac/" error="workflowtaskresults.argoproj.io is forbidden: User \"system:serviceaccount:argo:default\" cannot create resource \"workflowtaskresults\" in API group \"argoproj.io\" in the namespace \"argo\""
time="2023-02-16T21:39:08.057Z" level=info msg="Patch pods 403"

Usage of the "default" service account is discouraged as per the argo worklfows doc here. If you insist on using it - you need to grant it some privileges. You can do so by creating a role and then a rolebinding for the "service account". If you are using helm to install argo - it will create a service account with the appropriate role too. Just make sure you use it when you're submitting your workflow. Hope this helps

Upvotes: 0

Related Questions