CODEWITHSUNDEEP
hadoophdfs
abhiieor
abhiieor

Reputation: 3554

Creating multiple layer HDFS folder

For creating /user/SVLSTSLS/LostSales/sales-history-prep HDFS folder one can create HDFS folders incrementally like:

bash-4.1$ hadoop fs -mkdir /user/SVLSTSLS
bash-4.1$ hadoop fs -mkdir /user/SVLSTSLS/LostSales
bash-4.1$ hadoop fs -mkdir /user/SVLSTSLS/LostSales/sales-history-prep

Is there any way by which I can create final folder sales-history-prep as well as all intermediate folders (SVLSTSLS and LostSales), if not already present?

Upvotes: 1

Views: 3038

Answers (4)

XYZ
XYZ

Reputation: 382

You can use the FileSystem to create hdfs folders (as well the intermediate folders) in JAVA code.

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.*;
import org.apache.hadoop.fs.permission.FsPermission;
import org.apache.log4j.LogManager;
import org.apache.log4j.Logger;


Configuration hadoopConfiguration = new Configuration();
hadoopConfiguration.addResource(new Path("C:\\Xiang\\Softwares\\hadoop3.2\\hadoop-3.2.1\\etc\\hadoop\\core-site.xml"));



Path hdfs_P190415 = new Path("hdfs://localhost:9820/wimp/contract-snapshot/year=2019/month=4/day=15");
FileSystem fs = FileSystem.get(hadoopConfiguration)
fs.mkdirs(hdfs_P190415);

Then a folder (as well as the intermediate folders) will be created on the HDFS system. You can check under the terminal

hadoop fs -ls "/wimp/contract-snapshot/year=2019"

enter image description here

enter image description here

Before I run above code, the folder ~/month=4/day=15 does not exist. Now both the intermediate folder "~/month=4" and the final subfolder "~/month=4/day=15" are created.

The above JAVA code is tested on a standalone (windows) HDFS system, but should also work with multi clusters on a linux/production environment.

Upvotes: 0

maggi3066
maggi3066

Reputation: 1

hadoop fs -mkdir -p <paths>

-p option will create whole directory structure regardless parent directories were created or not.

You can provide multiple path to create, within single command by separating them with whitespace. For example,

hadoop fs -mkdir -p /fld1/fld11/fld111 /fld1/fld11/fld112

Upvotes: 0

Martin Milichovsky
Martin Milichovsky

Reputation: 730

hadoop fs -mkdir -p /user/SVLSTSLS/LostSales/sales-history-prep

From the documentation:

The -p option behavior is much like Unix mkdir -p, creating parent directories along the path.

Upvotes: 1

Dennis Jaheruddin
Dennis Jaheruddin

Reputation: 21563

It seems that you are looking for the -p flag.

If you use this flag, parent directories are also made if needed. Try it yourself

hadoop fs -mkdir -p  q/w/e/r

This should work, whilst the following will fail:

hadoop fs -mkdir  r/q/w/e

Upvotes: 3

Related Questions