In part 2 of this course, Barry Smart, Director of Data and AI, walks through a demo showing how you can use Microsoft Fabric to set up a "data contract" that establishes minimum data quality standards for data that is being processed by a data pipeline.
He deliberately passes bad data into the pipeline to show how the process can be set up to "fail elegantly" by dropping the bad rows and continuing with only the good rows. Finally, he uses the new Teams pipeline activity in Fabric to show how you can send a message to the data stewards who are responsible for the data set, informing them that validation has failed, itemising the specific rows that failed and the validation errors that were generated in the body of the message.
The demo uses the popular Titanic data set to show features in data engineering experience in Fabric, including Notebooks, Pipelines and the Lakehouse. It uses the popular Great Expectations Python package to establish the data contract and Microsoft's mssparkutils Python package to enable the exit value of the Notebook to be passed back to the Pipeline that has triggered it.
Barry begins the video by explaining the architecture that is being adopted in the demo including Medallion Architecture and DataOps practices. He explains how these patterns have been applied to create a data product that provides Diagnostic Analytics of the Titanic data set. This forms part of an end to end demo of Microsoft Fabric that we will be providing as a series of videos over the coming weeks.
00:12 Overview of the architecture
00:36 The focus for this video is processing data to Silver
00:55 The DataOps principles of data validation and alerting will be applied
02:19 Tour of the artefacts in the Microsoft Fabric workspace
02:56 Open the "Validation Location" notebook and viewing the contents
03:30 Inspect the reference data that is going to be validated by the notebook
05:14 Overview of the key stages in the notebook
05:39 Set up the notebook, using %run to establish utility functions
06:21 Set up a "data contract" using great expectations package
07:45 Load the data from the Bronze area of the lake
08:18 Validate the data by applying the "data contract" to it
08:36 Remove any bad records to create a clean data set
09:04 Write the clean data to the lakehouse in Delta format
09:52 Exit the notebook using mssparkutils to pass back validation results
10:53 Lineage is used to discover the pipeline that triggers it
11:01 Exploring the "Process to Silver" pipeline
11:35 An "If Condition" is configured to process the notebook exit value
11:56 A Teams pipeline activity is set up to notify users
12:51 Title and body of Teams message are populated with dynamic information
13:08 Word of caution about exposing sensitive information
13:28 What's in the next episode?
#microsoftfabric #dataengineering #greatexpectations #course #tutorial