Understanding metadata scalable data architecture book

Issue link: https://resources.zaloni.com/i/790575

Contents of this Issue


Page 12 of 22

bination of running specific algorithms over particular datasets, so it becomes extremely important to have metadata informa‐ tion about the intermediate processes, in order to enhance or improve it over time. Data lakes must be architected properly to leverage metadata and integrate with existing metadata tools, otherwise it will create a hole in organizations' data governance process because how data is used, transformed, and related outside the data lake can be lost. An incor‐ rect metadata architecture can often prevent data lakes making the transition from an analytical sandbox to an enterprise data platform. Ultimately, most of the time spent in data analysis is in preparing and cleaning the data, and metadata helps to reduce the time to insight by providing easy access to discovering what data is avail‐ able, and maintaining a full data tracking map (data lineage). A Modern Data Architecture—What It Looks Like Unlike a traditional data architecture driven by an extract, trans‐ form, load (ETL) process that loads data into a data warehouse, and then creates a rationalized data model to serve various reporting and analytic needs, data lake architectures look very different. Data lakes are often organized into zones that serve specific functions. The data lake architecture begins with the ingestion of data into a staging area. From the staging area it is common to create new/ different transformed datasets that either feed net-new applications running directly on the data lake, or if desired, feed these transfor‐ mations into existing EDW platforms. Secondly, as part of the data lake, you need a framework for captur‐ ing metadata so that you can later leverage it for various use case functionalities discussed in the previous section. The big data management platform of this modern data architecture can provide that framework. The key is being able to automate the capture of metadata on arrival, as you're doing transformations, and tying it to specific definitions like the enterprise business glossary. A Modern Data Architecture—What It Looks Like | 11

Articles in this issue

view archives of eBooks - Understanding metadata scalable data architecture book