Issues with attaching Spark configuration files using sparkConfigMap in YAML - kubeflow/spark-operator

Question

I'm using spark-operator provided by kubeflow (google spark-operator) to deploy spark applications on kubernetes.

I'm facing an issue while trying to attach Spark configuration files as a ConfigMap using sparkConfigMap in my YAML configuration. Despite following the documentation, the configuration doesn't seem to be applied correctly. However, when I explicitly specify the configurations using sparkConf, everything works as expected.

Here's a brief outline of what I'm doing:

I created a ConfigMap to hold my Spark configuration files:

apiVersion: v1
kind: ConfigMap
metadata:
  name: spark-property-files
data:
  spark-defaults.conf: |
    spark.driver.userClassPathFirst      true
    spark.executor.userClassPathFirst    true
    spark.eventLog.enabled               true
    spark.eventLog.dir                   file:///sparkfiles/spark-events/
    spark.history.fs.logDirectory        file:///sparkfiles/spark-events/
    spark.eventLog.rolling.enabled       true
    spark.eventLog.rolling.maxFileSize   128m

spark application

apiVersion: "sparkoperator.k8s.io/v1beta2"
kind: SparkApplication
metadata:
  name: spark-pi
  namespace: spark-operator
spec:
  type: Scala
  mode: cluster
  image: "nexus.example.com/spark:spark"
  imagePullPolicy: Always
  mainClass: org.apache.spark.examples.SparkPi
  mainApplicationFile: "local:///opt/spark/examples/jars/spark-examples_2.12-3.2.4.jar"
  sparkVersion: "3.0.1"
  restartPolicy:
    type: Never
  volumes:
    - name: nfs-volume
      persistentVolumeClaim:
        claimName: spark-pvc
  driver:
    cores: 1
    coreLimit: "1200m"
    memory: "512m"
    labels:
      version: 3.1.1
    serviceAccount: spark
    volumeMounts:
      - name: nfs-volume
        mountPath: "/sparkfiles/"
    nodeSelector:
      eks.amazonaws.com/nodegroup: "spark"
  executor:
    cores: 1
    instances: 1
    memory: "512m"
    labels:
      version: 3.1.1
    serviceAccount: spark
    volumeMounts:
      - name: nfs-volume
        mountPath: "/sparkfiles/"
    nodeSelector:
      eks.amazonaws.com/nodegroup: "spark"
  # sparkConf:
  #   "spark.driver.userClassPathFirst" : "true"
  #   "spark.executor.userClassPathFirst" : "true"
  #   "spark.eventLog.enabled": "true"
  #   "spark.eventLog.dir": "file:///sparkfiles/spark-events/"
  #   "spark.history.fs.logDirectory": "file:///sparkfiles/spark-events/"
  sparkConfigMap: "spark-property-files"

https://github.com/kubeflow/spark-operator/blob/master/docs/user-guide.md

i have tried to specify the event log location in properties in configmap, which didnt work.