Summary
Delta Lake is an open source storage layer available under Apache License 2.0
1
, while Databricks has announced Delta Engine, a new vectorized query engine that is 100% Apache Spark-compatible.
2
Delta Engine offers real-world performance, open, compatible APIs, broad language support, and features such as a native execution engine (Photon), a caching layer, cost-based optimizer, adaptive query execution, and dynamic runtime filters.
3
According to
Summary
Delta Lake is an open source storage layer that brings reliability to data lakes built on HDFS and cloud storage by providing ACID transactions. It addresses the common problems with data lakes, such as poor data quality, poor performance, and data versioning, and is available at delta.io to download and use under Apache License 2.0. Delta Lake is a great solution for data lakes that need reliable transactions, scalable metadata handling, data versioning, and a unified batch and streaming sink.
Open Sourcing Delta Lake - The Databricks Blog
databricks.com
Summary
Databricks has announced Delta Engine, a new vectorized query engine that takes advantage of modern CPU architecture and optimizations to Spark 3.0's query optimizer and caching capabilities. Delta Engine ties together a 100% Apache Spark-compatible vectorized query engine to take advantage of modern CPU architecture, making it easier for customers to adopt and scale a lakehouse architecture. Photon, a completely rewritten execution engine for Databricks, is the biggest innovation in Delta Engine, delivering performance improvements to all workload types while remaining fully compatible with open Spark APIs.
Delta Engine Introduction and Overview of How it Works -- The Databricks Blog
databricks.com
Summary
Delta Engine is a 100% Apache Spark-compatible query engine designed to deliver fast performance on modern cloud hardware for all data use cases across data engineering, data science, machine learning, and data analytics. It offers real-world performance, open, compatible APIs, broad language support, and features such as a native execution engine (Photon), a caching layer, cost-based optimizer, adaptive query execution, and dynamic runtime filters. With Delta Engine, users can bring even better performance to their Delta Lake on Databricks.
Delta Engine - Databricks
databricks.com
Explanation and details on Databricks Delta Lake. ... It is a Spark proprietary extension and cloud-only. It has been open sourced and the code can be found ...
Databricks Delta Lake | James Serra's Blog
jamesserra.com
