CODEWITHSUNDEEP
databricksazure-databricks
learner
learner

Reputation: 1023

Does Azure Databricks and Delta Layer make it a Lakehouse?

Even after going through many resources, I have failed to understand what constitutes a lakehouse, hence my question below.

If we have Azure Gen 2 Storage, ADF, and Azure Databricks with the possibility of converting the incoming CSV files into Delta tables can that be called a "Lakehouse" architecture or is it called a "Delta Lake"?

Or is it the "SQL analytics" engine over and above the Delta Lake layer that makes it a "Lakehouse"?

Please clarify.

Upvotes: 1

Views: 249

Answers (1)

bpowell
bpowell

Reputation: 26

At a high level a Lakehouse must contain the following properties:

  1. Open direct access data formats (Apache Parquet, Delta Lake etc.)
  2. First class support for machine learning and data science workloads
  3. state of the art performance

Databricks is the first Lakehouse because it meets the above three properties. Specifically, if you are using Databricks with ADLS and converting all your data (json, csv, parquet, messages etc.) into Delta tables that are available within Databricks. Then that is the making of a Lakehouse, but it still needs to be built and supported. The Databricks platform allows us to satisfy points 2 and 3 above and Delta Lake satisfies 1 ad 3 (performance relies on the engine and the storage which is why 3 is mentioned twice).

Leveraging Databricks and accessing data stored in Delta is a Lakehouse. By adding Databricks SQL (formally SQL Analytics) we allow more users to access and use the Lakehouse. In Databricks SQL users are using the same compute and data as the data engineer does in Databricks, they just have a different UI that they are familiar with. Additionally, Databricks SQL is optimized for SQL and BI workloads while the notebook environment is better for engineering and data science

As a fun read you should check our the Lakehouse whitepaper.

Upvotes: 1

Related Questions