CODEWITHSUNDEEP
apache-sparkapache-spark-sqldatabricks
YFl
YFl

Reputation: 1429

describe spark table/view comment

We can create a table and view it with a comment describing it. For example (from spark docs):

CREATE TABLE student (id INT, name STRING, age INT) USING CSV
    COMMENT 'this is a comment'
    TBLPROPERTIES ('foo'='bar');

How can you retrieve the comment in a "clean format"? By clean format, I mean only (or almost only) the table name and the comment describing it. Any other solution I've found bloats me with all the column types and information (which I don't need in this case).

I've tried:

DESCRIBE student
DESCRIBE EXTENDED student
SHOW CREATE TABLE student
DESCRIBE DETAIL student -- databricks only
SHOW VIEWS FROM default -- to try seeing views description
SHOW TABLES FROM default -- to try seeing tables description

The best would be to have something like SHOW TABLE/SHOW VIEWS, but with a column adding the description.

Is there an out-of-the-box solution for this? If not, is there a good custom way to achieve it?

Thank you.

Upvotes: 1

Views: 4379

Answers (1)

Kombajn zbożowy
Kombajn zbożowy

Reputation: 10703

There is no way to get table comment only. However it's fairly easy to filter it out of DESCRIBE TABLE statement using Scala.

spark.sql("CREATE TABLE student (id INT, name STRING, age INT) USING CSV COMMENT 'this is a comment'")
spark.sql("DESCRIBE TABLE EXTENDED student").filter($"col_name" === "Comment").show

+--------+-----------------+-------+
|col_name|        data_type|comment|
+--------+-----------------+-------+
| Comment|this is a comment|       |
+--------+-----------------+-------+

Upvotes: 2

Related Questions