Tech Talk Thursday: Build a Synapse Pipeline Part 3: Full and Incremental Refresh / Building a Notebook

Welcome to Part Three of this three-part video series on creating a pipeline to handle both full and incremental refreshes in Azure Synapse presented by OmniData’s Principal Solutions Architect, Ray Marquina.

In Part One, we created our control table using Delta Lake. 

In Part Two, we left off with building our initial pipeline and ended with loading our port and vessel tables.

In this video, we will finish up with the tutorial on bringing over your full and incremental refreshes.

• Use expression builder in our pipelines to drive which select statement we want to run
• Modify our file name to differentiate between full and incremental loads
• Build a notebook to perform an update to the IS initial load flag and watermark after the data has been loaded

Make sure you watch all three videos in this informative series!