CODEWITHSUNDEEP
databricksazure-databricks
Carltonp
Carltonp

Reputation: 1344

Databricks file copy with dbtuils only if file doesn't exist

I'm using the following databricks utilites (dbutils) command to copy files from one location to another as shown below:

dbutils.fs.cp('adl://dblake.azuredatalakestore.net/jfolder2/thisfile.csv','adl://cadblake.azuredatalakestore.net/landing/')

However, I want the file to be copied over only if no such file with the same name 'thisfile.csv' exists.

Can someone let me know whether is that possible?

If not, is there any other workaround?

Upvotes: 3

Views: 23873

Answers (1)

Foxan Ng
Foxan Ng

Reputation: 7151

dbutils.fs.ls() lists files in the given path.

ls

So you can check if thisfile.csv exists before copying the file:

if "thisfile.csv" not in [file.name for file in dbutils.fs.ls("adl://cadblake.azuredatalakestore.net/landing/")]:
    dbutils.fs.cp("adl://dblake.azuredatalakestore.net/jfolder2/thisfile.csv", "adl://cadblake.azuredatalakestore.net/landing/")

Upvotes: 7

Related Questions