Alter table rename column on Hudi tables on AWS EMR and AW S3

Question

Using EMR 7.2

[root@ip-172-31-34-128 ~]# spark-sql --jars /usr/lib/hudi/hudi-spark-bundle.jar,/usr/lib/hudi/hudi-aws-bundle.jar --conf "spark.serializer=org.apache.spark.serializer.KryoSerializer" --conf "spark.sql.catalog.spark_catalog=org.apache.spark.sql.hudi.catalog.HoodieCatalog" --conf "spark.sql.extensions=org.apache.spark.sql.hudi.HoodieSparkSessionExtension" --conf spark.sql.hive.convertMetastoreParquet=false
Aug 14, 2024 2:21:58 AM org.apache.spark.launcher.Log4jHotPatchOption staticJavaAgentOption
WARNING: spark.log4jHotPatch.enabled is set to true, but /usr/share/log4j-cve-2021-44228-hotpatch/jdk17/Log4jHotPatchFat.jar does not exist at the configured location

Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
24/08/14 02:22:01 WARN HiveConf: HiveConf of name hive.server2.thrift.url does not exist
24/08/14 02:22:04 WARN Client: Neither spark.yarn.jars nor spark.yarn.archive is set, falling back to uploading libraries under SPARK_HOME.
Spark Web UI available at http://ip-172-31-34-128.us-west-2.compute.internal:4040
Spark master: yarn, Application Id: application_1723602092910_0001
spark-sql (default)> show tables;
account
Time taken: 1.783 seconds, Fetched 1 row(s)
spark-sql (default)> ALTER TABLE account RENAME COLUMN creation_date TO creation_date2;
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
[UNSUPPORTED_FEATURE.TABLE_OPERATION] The feature is not supported: Table `spark_catalog`.`default`.`account` does not support RENAME COLUMN. Please check the current catalog and namespace to make sure the qualified table name is expected, and also check the catalog implementation which is configured by "spark.sql.catalog".
spark-sql (default)>