CODEWITHSUNDEEP
hadoophive
Metaphor
Metaphor

Reputation: 6405

How to ignore comment lines on load

I'd like to load IIS weblogs into a Hive table but the comments in the log file are getting in the way. Is there a way to have Hive's Load ignore lines starting with #?

Upvotes: 0

Views: 145

Answers (1)

Urvishsinh Mahida
Urvishsinh Mahida

Reputation: 1450

If your Table schema has column for whole "entry" as STRING, then you can put a condition in WHERE clause of your query as WHERE substr(entry, 1, 1) != "#"

Hive is schema on read and thus wont complain about the whole record as long its able to fetch the first character.

You may want to prune such record into a cleaned table, so that who so ever queries the table does not to put the WHERE condition of substr(entry, 1, 1) != "#"

Upvotes: 1

Related Questions