Databricks spark.readstream format differences

Question

I am having confusion on the difference of the following code in Databricks

spark.readStream.format('json')

vs

spark.readStream.format('cloudfiles').option('cloudFiles.format', 'json')

I know cloudfiles as the format would be regarded as Databricks Autoloader . In performance/function comparison , which one is better ? Anyone has some experience on that?

Thanks

Answer 1

In my experience, for scheduled batch jobs I have found spark streaming to be ~60% faster on average. You can use autoloader for realtime event based updates, as between events it wont use compute (i.e. you do not need to poll event source continuously)

Ref data of a batch job:

Spark read/writestream - 8412 rows in 1min 52 secs
Autoloaded - 8412 records in 5 min 30 seconds

Answer 2

There are multiple differences between these two. When you use Auto Loader you get at least, there are more things (see doc for all details):

• Better performance, scalability and cost efficiency when discovering new files. You can either use file notification mode (when you get notified about new files using cloud-native integration) or optimized file listing mode that uses native cloud APIs to list files and directories. Spark's file streaming is relying on the Hadoop APIs that are much slower, especially if you have a lot of nested directories and a lot of files
• Support for schema inference and evolution. With Auto Loader you can detect changes in the schema for JSON/CSV/Avro, and adjust it to process new fields.