Real-World Data Movement and Orchestration Patterns using Azure Data Factory V2
Ғылым және технология
Links:
Sign up to our Meetup group www.meetup.com/hybrid-virtual...
Join our LinkedIn group / 12487962
Follow us on Twitter / hybridvug
...and of course don't forget to subscribe to our channel, like the video, and hit the bell! It really helps us in our mission to produce free content for the masses.
Session Details:
In this session, we will start with an overview of Azure Data Factory V2 concepts, then show you how you can use metadata to quickly build scalable serverless pipelines to move data from disparate data sources including On-Premises and Platform As A Service.
Next, we will look at how to integrate the solution using continuous integration and deployment techniques.
Finally, we will look at how to schedule, monitor, and log our solution.
Whether you are just getting started with Azure Data Factory or looking to make your current data factory robust and enterprise-ready this session will take you to the next level.
Speaker Bio:
Jason Horner, Global Thought Leader and International Man of Leisure
Hi, my name is Jason. I spend most of my day helping clients solve business problems mostly in the Data and Advanced Analytics spaces. Sometimes this involves various and sundry cloud technologies including Azure Data Factory, Azure SQL Data Warehouse, Azure Data Lake Store, Azure Databricks, HDInsight and Azure SQL Database. I'm fluent in several languages including: SQL, C#, Python, and PowerShell.
I'm a Microsoft Certified Master of SQL Server (MCM) and have been recognized for my technical excellence and evangelism efforts by Microsoft by being awarded the Most Valuable Professional (MVP) for the last 5+ years.
In my off hours I like to snowboard, karaoke, ride roller coasters, and play arcade games.
Host/s:
Mark Broadbent.
---
Originally recorded on 14th July 2020.
Video Production: Mark Broadbent 24th August 2020.
Пікірлер: 6
Hi, where can I find the samples templates?
@HybridVirtualGroup
2 жыл бұрын
Hope this is what you are looking for Diego docs.microsoft.com/en-us/azure/data-factory/solution-templates-introduction
Sir , i need training for Azure Data Engineer please suggest 🙏
Merge and Incremental load explanation is bit confusing. Would you please bit elaborate on it
@BigBossInd7236
Жыл бұрын
Merge approach is used if you don't have a delta identifier in the source table e.g. created and updated dates. In that situation, form a hash-key using all non-key columns in a source table. Then using the primary key to match source and target table row and comparing hash-key to decide whether source record is changed or not. If primary key matches and hash-key doesn't then either insert a new updated record or update existing target record. If the primary key doesn't match then insert in target. There is a situation when the primary key doesn't match, either source record doesn't exist in target or it exists in the target but got deleted from the source. You need to handle this accordingly. Incremental is simple, it means you have a delta identifier in the source. Initially you will load all the records from the source into the target table and save maximum of either the created or updated date in the control table for each source table. So, next time you load data, pick the records after that maximum date you have in the control table and keep on repeating it. Hope you have understood it 🙂
@shrutichikode8059
2 ай бұрын
@@BigBossInd7236 great explanation, thank you :)