CODEWITHSUNDEEP
hadoophive
Dolan Murvihill
Dolan Murvihill

Reputation: 345

Hive - column type name too long

I want to use rcongiu's hive-json-serde to store non-trivial JSON documents complying with an open standard. I've used Michael Peterson's convenient hive-json-schema generator to produce a CREATE TABLE statement that should work, except for its size.

The JSON documents I am encoding follow a well-defined schema, but the schema contains maybe a hundred fields, nested up to four levels deep. A Hive column type that matches the standard is very, very long (around 3700 characters), and when I run my generated create table statement I get the error

FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask.
InvalidObjectException(message:Invalid column type name is too long: <the
really long type name>)

The statement looks like this:

CREATE TABLE foobar_requests (
  `event_id` int,
  `client_id` int,
  `request` struct<very long and deeply nested struct definition>,
  `timestamp` timestamp)
ROW FORMAT SERDE 'org.openx.data.jsonserde.JsonSerDe';

Any path forward to storing these documents?

Upvotes: 2

Views: 3557

Answers (3)

Pranav Pahuja
Pranav Pahuja

Reputation: 9

This issue occurs, when the name of one of the Column Type is longer than the default of 2000 characters. Solution: To resolve this issue, do the following: 1.Add the following property through Ambari > Hive > Configs > Advanced > Custom hive-site: hive.metastore.max.typename.length=10000

The above value is an example, and it needs to be tuned according to a specific use case.

2.Save changes, restart services and recreate the table.

Upvotes: 0

Avinash Nishanth S
Avinash Nishanth S

Reputation: 564

Add the following property through Ambari > Hive > Configs > Advanced > Custom hive-site:
hive.metastore.max.typename.length=14000

Upvotes: 3

Daniel DC
Daniel DC

Reputation: 21

Hive has a problem with very long column definitions. By default the maximum number of chars supported is 4000 so if you really need more than this you'll have to alter the metastore database by extending the length of COLUMNS_V2.TYPE_NAME.

If you like to read more about the issue go to this link:

https://issues.apache.org/jira/browse/HIVE-12274

Upvotes: 2

Related Questions