Introducing Delta Live Tables

By Author: Published by rajaniesh
Total Articles: 4
Comment this article

Also, in another previous blog, I also have given a glimpse at its implementation through Delta Tables in Azure Databricks. The Delta Table is a building block of designing Data Pipeline for Delta Lake. The Delta Lake is an open-source project aimed to implement Lake House Architecture with pluggable components for Storages and Computes. It is necessary to recall the concepts of Delta Lake before understanding Delta Live Tables. I hope the following brief discussion on Delta Lake will help to serve the purpose.

Delta Lake is a standard, offering ultimate solution to process Batch/Historical data and Stream/Real-time data in a same data pipeline without compromising on simplicity of solution, but a boon with data integrity (Which is a serious bottleneck in implementing Lambda Architecture), Open Formats, Delta Sharing (Industry’s first open protocol to secure sharing of data across tools, applications, organizations, hardware without using any staging environment. Please refer to my earlier blog on ‘Delta Lake’ for more details). The features of Delta Lake improve both the manageability and performance of working ...
... with data in storage objects and enable a “Lake House” paradigm that combines the key features of data warehouses and data lakes augmented with High Performance, Scalability and cost economics.

Maintaining Data Quality and Reliability at large scale has always been a complex issue all over the industry. If one pipeline fails, all depending on pipelines at downstream fails. Operational complexities dominate focus on development. For it till the date, many Data Workflow Management solutions have been proposed. Most of the solutions may be failing in offering an absolute solution for a data workflow in managing Batch and Stream Processing pipelines together. Industry has been continuously undergoing the pain of concerns around the Workflow Management. These concerns can be summarized in mainly following 3 points.

Complex Pipeline Development
Difficult to switch between batch and stream pipelines
Hard to build and maintain table dependencies
Systems short of supporting Incremental workloads with partitions over time period
Poor Data Quality
Difficult to monitor and checks for data quality on Formulas, Rules Constraints, Value Ranges
Insufficient support for enforcing data quality using simple approach.
Difficult to trace Data Lineage.
Operational concerns
Silos within teams
Difficult to check and maintain data operations because of poor observability at the data granularity level.
Error Handling, Recovery, reload is laborious
Support for version control with branching and merging
Data Governance with Data Confidentiality, Data Access Control with Masking/Encryption.