Build a Synapse Pipeline Part 3: Full and Incremental Refresh / Building a Notebook
Welcome to Part Three of this three-part video series on creating a pipeline to handle both full and incremental refreshes in Azure Synapse presented by OmniData™’s Principal Solutions Architect, Ray Marquina.
In Part One, we created our control table using Delta Lake.
In Part Two, we left off with building our initial pipeline and ended with loading our port and vessel tables.
In this video, we will finish up with the tutorial on bringing over your full and incremental refreshes.
- Use expression builder in our pipelines to drive which select statement we want to run
- Modify our file name to differentiate between full and incremental loads
- Build a notebook to perform an update to the IS initial load flag and watermark after the data has been loaded
Make sure you watch all three videos in this informative series!
OmniData is a leading Microsoft partner and professional services firm specializing in data and analytics modernization and AI solutions. We provide actionable insights and scalable solutions powered by AI, seamlessly integrating with existing systems to maximize efficiency. Our team of veteran specialists excels in solution architecture, data engineering, and business intelligence. We thrive on partnering with companies across various industries and global markets, helping them quickly conceptualize and address complex business data challenges.