Data Lineage In Etl And Elt Solution

By Author: lumendata
Total Articles: 12
Comment this article

Enterprises have built massive data infrastructures to capture and manage their ever-growing mountains of data. But as data stores increase, the pipelines that carry precious information to the business become murkier, making the resulting data analysis less trustworthy. Informatica’s Enterprise Data Catalog (EDC) can help you shed light on data transformations along your pipelines, and LumenData has further built tools to extract and visualize additional data lineage to extend the use of Informatica EDC. Here’s how.

More Data Leads to More Transformations
All organizations are data-driven. Complex data engineering and rapid development to accomplish this has brought a new challenge to every CDO/CIO: How do we manage and visualize the data movement across the organization? This challenge is exacerbated with the advent of massive data warehouses, where the tendency is to store all enterprise data “just in case we need it in the future.” Instead of Extract (from source), Transform (to an intelligible trustable form), and then Load (to the warehouse) — otherwise known as ETL — many organizations ...
... extract data and dump it into a warehouse with the intention of transforming it when they need it, what is known as ELT.

When data is required for a particular analytic purpose, enterprise developers read the raw data from the warehouse, transform it to the form needed, and store it back in the warehouse or push it to another destination. They use a technique called data pipelines to perform these calculations (“transformations”), reading from one data set, applying algorithms on the data, and storing in the intermediate form in the warehouse. Then repeat this with more transformations. They do this as many times as needed until the final data is in the desired form.