CODEWITHSUNDEEP
hadoophivehdfs
user10591810
user10591810

Reputation: 49

How to create a table on top of HDFS folders with subfolders and multiple csv files

We have a hdfs folder with structure like /data/year/day/.csvfiles, so we have multiple csv files stored per day in hdfs folders e.g. /finance/2019/20190101/ multiple csv files. Similarly, there will be 365 folders in /finance/2019/ for 365 days of a year. I would like to create a Hive table on top of /finance/2019/ so I can query all data, however my create table only works if I specify specific folder such as /finance/2019/20190101/, and then query returns resultset. If I try to create table with location as /finance/2019/ the table cannot retrieve any data and the result is always 0.

Upvotes: 0

Views: 337

Answers (1)

notNull
notNull

Reputation: 31460

To read data from subdirectories set these properties!

hive> SET hive.mapred.supports.subdirectories=TRUE;
hive> SET mapred.input.dir.recursive=TRUE;

Try to select the data from the hive table again.

In case if you don't want to set these properties everytime in hive shell then add these properties in your hive-site.xml file

<property>
    <name>mapred.input.dir.recursive</name>
    <value>true</value>
  </property>

  <property>
    <name>hive.mapred.supports.subdirectories</name>
    <value>true</value>
  </property>

Upvotes: 1

Related Questions