A well-known consumer packaged goods (CPG) company specializing in pet food asked Imaginet to perform maintenance on their existing data collection processes. We work in a staff augmentation role for this client and do ongoing work for them. Their data departments are divided into different business areas (we will refer to them as teams). On this project, we were working with two of their teams. Both teams required general maintenance and updates to their data collection tools – essentially a data collection upgrade.

Team 1:

Their primary focus is collecting and processing data about pet foods and products (nutritional claims they make, calories, ingredients, product history, etc.). This process to collect and process the data about pet food had not been touched or updated for years, and there were many points in the process where it required manual intervention, and it was difficult and time-consuming to add data from another source. The system was using outdated functionality that was difficult to maintain, and our client was in desperate need of a data collection upgrade.

This was an example of addressing technical debt that had accumulated in this system. Technical debt is the difference between how code is written, and the optimal way code can be written. Often technical debt originates due to time constraints or lack of knowledge, but it accumulates over time and is often not given enough attention. As it accumulates, technical debt becomes increasingly difficult to fix, and makes the process of writing new code for new functionality slower and more prone to errors.

In this case, the existing outdated code needed to be updated to use their current, more modern, platforms. We also made changes that would allow for the incorporation of new data sources more easily.

Reducing your technical debt is essential – you can read more about how to reduce it here.

Team 2:

Product pricing is a complex process that has many variables and must be flexible to accommodate changes quickly and easily. Setting new prices usually involves experimenting with many “what-if” scenarios before deciding on a future pricing strategy. Communicating pricing data across a large enterprise is also critical to ensuring that production, marketing, and sales are working from the same data. This team had a process that was prone to manual errors and data quality issues, especially when creating a new set of prices.

The pricing system relied on Excel spreadsheets and SQL database tables, so we improved the loading process of the SQL tables from the spreadsheets. We made database changes by restructuring and properly normalizing the tables, removing redundant columns, and enforcing referential integrity. This significantly reduced the data quality issues that had been occurring. In the Power App they had built to help them create new pricing data for what-if scenarios, we fixed bugs and made enhancements to make it more user-friendly and reliable.

Their data reports were outdated, so we designed a dimensional model in the SQL database and built a well-structured Power BI semantic model. Additionally, we performed work on the data loading to simplify the process of transferring data into the system. Now, the data is sourced directly from the tables connected to Power Apps.

We removed reports that were no longer needed and created reports and dashboards our client wanted.

For other departments, there are two places where they can get the correct pricing data – most of the departments consume the pricing data from Power BI Data Flows that were enhanced during this effort. We also persisted the pricing data into the Data Lake using Azure Databricks notebooks and Azure Data Factory, which is the new preferred data architecture. Both the Power BI Data Flows and the Data Lake tables have access controls configured to ensure proper security of this data. By publishing this critical pricing data in these two locations, other departments have access to the correct and up to date data as they need, which allows business decisions to be made on trustworthy data.

Why This Project Was Unique

We have been working with this client for many years and have learned their processes and preferred architectures. Our team feels integrated with their organization and has come to understand how they operate. They trust our work and we collaborate well, offering our ideas and expertise to help them continually improve and optimize their processes.

Maintaining and building client relationships is important to Imaginet. We ensure we are working with you and including you in the process. We approach each project as a partnership and work closely with our clients.

Our team has been working hard these last few months, and we are excited to share our success with you. Don’t forget to subscribe to our newsletter to stay updated on our most recent Microsoft 365 projects. Fill out the form at the bottom of the page if your data is unreliable, you want to upgrade your data factory, or you need a data collection upgrade.