June 2019 Discovery Hub Release

This week, TimeXtender released a new version of Discovery Hub, I have been playing around with the preview for a last several weeks, so wanted to share my impression. As far as quality-of-life features for the Discovery Hub developer goes, this is a huge release. If you haven’t upgraded for a while, now is a good time to consider doing so.

Revamped User Interface

I am so excited that we now have a new User Interface to work with. This has been something TimeXtender has wanted to do for a very long time, I remember seeing a prototype back in early 2013.

There are now three permanent components in the UI, to the very left we have the solution explorer. To the very right is the data movement pane, which will be empty when no object with a data movement or transformation context is selected. Most importantly, in the center is the main working area, where you can open any number of tabs, grouping them as needed.

This new UI utilizes the full screen, giving a much better flow in the work by dragging elements between tabs, I generally set it up from the left to the right. If you are not used to doing it, now is the time to start using Discovery hub in full screen.  

Support for Azure Data Lake Gen 2 and Azure Databricks

On the storage side, the ODX Server adds support for Azure Data Lake Store Gen 2 in addition to the existing support for SQL on-premises, Azure SQL (MI) and Azure Data Lake Store Gen 1.

On the data movement side Azure Data Lake Analytics is replaced with Azure Databricks for handling advanced incremental load (updates and deletes), incremental to data warehouse and selection rules. Importantly, Databricks can now handle the transfer between Azure Data Lake Store and the Data Warehouse, reducing the need for the Discovery Hub application to directly interact with the data to only the initial load from source to ODX.

This is a significant move towards a fully managed PaaS experience with Discovery Hub.

Improved Upgrade Experience

This is a nice little bonus, when you install the new version and activate your license, you are prompted with a dialog that allows you to import settings from a previous version of Discovery Hub. The wizard also takes you through connecting to your ODX Server, upgrading the repository and reminds you to update the settings of the windows services. While we are waiting for a proper in-place upgrade this is a good step forward.

You still have to remember to configure the correct repository settings for your service user after the upgrade.

If you did not already upgrade to version 19.2.x, you will also be taken to the performance recommendations wizard once you open your first project. This is a great tool that helps make sure you take advantage of the performance improvements that have been released over the last many years. If you have been using Discovery Hub for many years and have not been very methodical about implementing the new performance features, this could give you a lot of potential gains.

Keep lookups up to date

A new feature was released in beta in version 18.10 (that is October 2018, by the way), allowing you to keep lookups up-to-date for tables that are incrementally loaded. This means that when you perform a lookup from a source table, the lookup value in the destination table would be updated even when the destination record did not change. This was achieved by clever use of hash values to detect when an update of the destination table was needed.

With the original implementation, certain changes in the source table could trigger an unnecessary full load of the destination table. The final implementation in this version has addressed this issue and slightly simplified the configuration (the entire row hash-key is gone, and so is the check-box for it – we won’t miss it).

This is a great feature that addresses an issue that is sometimes overlooked with annoying consequences down the road, and it performs well even with large amounts of data.

Other updates

I also want to plug a small change here as the battle for the right way to create lookup joins is finally over, the UI will now show the relationship used directly on the lookup, even when no explicit relationship is defined. See the screenshot.

Reach out to me if you want to talk about a plan to upgrade to the new version.

-Tobias Eld

omnidata author tobias eld

Tobias Eld

Co-Founder & CTO

As Co-founder and CTO, Tobias runs OmniData’s Azure Practice, covering Analytics, Architecture and Cloud Migration. With twenty years of experience in information technology, Tobias is expanding on his former tenure running the North American TimeXtender consulting practice. His expertise extends deeply into the entire Microsoft Data Stack, both cloud and on-premises. We are also applications experts in countless top data platforms including Dynamics and many others. We are experts because of Tobias' leadership and mentorship.