A provider of real estate advisory services contacted Imaginet for support on a system we had built for them a few years ago (mostly built using Azure SQL). They have multiple databases that have monthly costs associated with them. Our client was looking for a cheaper solution as the monthly costs were quite significant. They also had an existing capacity agreement with Databricks, so our goal was to conduct a Databricks migration with our clients databases.  

What We Did 

We started from scratch and built all the coordinates and objects for our client’s architecture. We used metadata to create their tables and developed a custom autoloader to transfer files to their destination tables, utilizing Databricks-driven technology in the backend.  

To standardize the data deployment strategy, we built Databricks bundles. As part of our delivery, we reconciled with their existing SQL sources. This provides our client with the ability to compare the results from the data in Databricks to their previously existing sources. Furthermore, we compiled all their data into a single database, making it more easily accessible.  

Why Our Solution Was Helpful  

Our solution caused our client to take advantage of their partnership with Databricks, which allowed them to save money. Having their data together in one location is more convenient for our client and makes it easier to organize and access data. Additionally, we built jobs for them, removing the need for a data factory – another benefit of the data being in one location. 

Why This Project Was Unique  

This was Imaginet’s first time working with Autoloader in a Databricks environment. We were able to quickly learn and adapt and provided the client with a fully customized solution that could address their business needs.  

Our client was also new to Autoloader, and they were impressed with the convenience and support offered by the solution. Overall, our customization of Autoloader will allow for additional, easy transformations. Thus, eliminating the need for multiple transformations, which is what they were experiencing with their previous Azure SQL environment.  

Our team has been working hard these last few months, and we are excited to share our success with you. Don’t forget to subscribe to our newsletter to stay updated on our most recent Microsoft 365 projects. Fill out the form at the bottom of the page if you think a Databricks migration is right for your organization.