A well-known consumer packaged goods (CPG) company specializing in pet food asked Imaginet to perform maintenance on their existing data collection processes. We work in a staff augmentation role for this client and do ongoing work for them. Their data departments are divided into different business areas (we will refer to them as teams). On this project, we were working with two of their teams. Both teams required general maintenance and updates to their data collection tools – essentially a data collection upgrade.
Team 1:
Their primary focus is collecting and processing data about pet foods and products (nutritional claims they make, calories, ingredients, product history, etc.). This process to collect and process the data about pet food had not been touched or updated for years, and there were many points in the process where it required manual intervention, and it was difficult and time-consuming to add data from another source. The system was using outdated functionality that was difficult to maintain, and our client was in desperate need of a data collection upgrade.
This was an example of addressing technical debt that had accumulated in this system. Technical debt is the difference between how code is written, and the optimal way code can be written. Often technical debt originates due to time constraints or lack of knowledge, but it accumulates over time and is often not given enough attention. As it accumulates, technical debt becomes increasingly difficult to fix, and makes the process of writing new code for new functionality slower and more prone to errors.
In this case, the existing outdated code needed to be updated to use their current, more modern, platforms. We also made changes that would allow for the incorporation of new data sources more easily.
Reducing your technical debt is essential – you can read more about how to reduce it here.
Team 2:
Product pricing is a complex process that has many variables and must be flexible to accommodate changes quickly and easily. Setting new prices usually involves experimenting with many “what-if” scenarios before deciding on a future pricing strategy. Communicating pricing data across a large enterprise is also critical to ensuring that production, marketing, and sales are working from the same data. This team had a process that was prone to manual errors and data quality issues, especially when creating a new set of prices.
The pricing system relied on Excel spreadsheets and SQL database tables, so we improved the loading process of the SQL tables from the spreadsheets. We made database changes by restructuring and properly normalizing the tables, removing redundant columns, and enforcing referential integrity. This significantly reduced the data quality issues that had been occurring. In the Power App they had built to help them create new pricing data for what-if scenarios, we fixed bugs and made enhancements to make it more user-friendly and reliable.
Their data reports were outdated, so we designed a dimensional model in the SQL database and built a well-structured Power BI semantic model. Additionally, we performed work on the data loading to simplify the process of transferring data into the system. Now, the data is sourced directly from the tables connected to Power Apps.
We removed reports that were no longer needed and created reports and dashboards our client wanted.
For other departments, there are two places where they can get the correct pricing data – most of the departments consume the pricing data from Power BI Data Flows that were enhanced during this effort. We also persisted the pricing data into the Data Lake using Azure Databricks notebooks and Azure Data Factory, which is the new preferred data architecture. Both the Power BI Data Flows and the Data Lake tables have access controls configured to ensure proper security of this data. By publishing this critical pricing data in these two locations, other departments have access to the correct and up to date data as they need, which allows business decisions to be made on trustworthy data.
![Data collection upgrade](https://imaginet.com/wp/wp-content/uploads/2024/04/Purina-CSI-Screenshot-1.png)
![Data collection upgrade](https://imaginet.com/wp/wp-content/uploads/2024/04/Purina-CSI-Screenshot-1-1.png)
Why This Project Was Unique
We have been working with this client for many years and have learned their processes and preferred architectures. Our team feels integrated with their organization and has come to understand how they operate. They trust our work and we collaborate well, offering our ideas and expertise to help them continually improve and optimize their processes.
Maintaining and building client relationships is important to Imaginet. We ensure we are working with you and including you in the process. We approach each project as a partnership and work closely with our clients.
Our team has been working hard these last few months, and we are excited to share our success with you. Don’t forget to subscribe to our newsletter to stay updated on our most recent Microsoft 365 projects. Fill out the form at the bottom of the page if your data is unreliable, you want to upgrade your data factory, or you need a data collection upgrade.
Technologies Used
- Power Apps
- Power BI
- Azure Databricks
- Azure Data Factory
- Azure DevOps
- SQL Server
- Oracle
![Power BI Data Model](https://imaginet.com/wp/wp-content/uploads/2023/03/Power-BI.webp)
discover more
Secret Manager to Protect Sensitive Data in App Development September 5, 2024 Secret Manager to Protect Sensitive Data in App Development How you deal with sensitive data is critical to your organization’s success. If you are not careful with sensitive…
Protecting Sensitive Data: Ensuring Privacy from a Data Engineering Perspective August 29, 2024 Protecting Sensitive Data: Ensuring Privacy from a Data Engineering Perspective In a bank call center, staff members handle clients’ data. They have to identify clients for various…
Custom Script Setting in SharePoint & OneDrive Removed August 22, 2024 Last week, we published a blog discussing Microsoft retiring SharePoint Add-Ins. In that blog, we briefly discussed how the Classic Mode for Custom Scripts has been affected by that…
Let’s Build Something Amazing Together
From concept to handoff, we’d love to learn more about what you are working on.
Send us a message below or drop us a line at 1-800-989-6022.