Reputation: 1914
Normally, I can add a column description with transform
like this:
from transforms.api import Input, Output, transform
from utils import COLUMN_DESCRIPTIONS
@transform(
output=Output("/Shared/output"),
raw=Input("/Shared/raw_input")
)
def clean_table(raw, output):
raw = raw.dataframe()
output.write_dataframe(
raw,
column_descriptions=COLUMN_DESCRIPTIONS
)
Can I do a similar thing with transform_df
?
Upvotes: 2
Views: 497
Reputation: 1816
Unfortunately it's not currently possible to output column descriptions with transform_df
, you have to use the transform
decorator for this. This is because transform_df
returns a native Spark DataFrame, which doesn't support column descriptions.
More information about column descriptions is available in the Palantir docs on column metadata.
Upvotes: 1