Last week, I sat down with one of our data experts, Olena Shevchenko, to get her thoughts on clean data and why it’s important. As someone with a non-technical background, I wanted to understand the steps you can take to ensure you are working with clean data.  

Clean Data: Step 1 

Olena explained the first step she would take is exploring. She gave the example of having a very wide data table, where you may or may not know the source of the data. You want to explore what kind of naming standards there are for the columns and understand what the underlying data behind it is. You also want to understand how full your data is (or how many empty values there are).  

You’ll want to analyze the quality of every column and row separately. If you find your rows have only, say, 10% of the expected data available, chances are you don’t have enough useable data. If this is the case, you won’t be able to use this data to make predictions or decisions and therefore it would be a good candidate for being thrown away.  

Throwing away data isn’t always in option. Especially if there is a limited amount of data for a particular record. In this case, another option is to look at how many types of records you have and separate them in their own tables or groups.  

Clean Data: Step 2 

The second step goes deeper into the exploratory data analysis realm. You want to analyze the data type (categorical, numerical, or binary data), and then for each type you want to analyze the distribution behind the data. For example, if you are analyzing binary data, you may want to determine how balanced the true and false is. For numerical data, you may want to see if the distribution is normal, skewed, flat, etc. You’ll also want to determine if there are any outliers.  

You want to understand these patterns and the expected behaviour of your current model for future modelling.  

Clean Data: Step 3

You also want to determine if there are any records that are different to the rest of the representations of your data. You’ll have to determine if there’s a way to validate it, but you’ll also need to research why it’s so different. 

Additionally, Olena discussed the importance of eliminating duplicate records. Duplicate records can lead to inaccurate data analysis and reporting and can affect your data’s integrity.  

Clean Data: An Outliers Example  

Olena provided a simple, yet effective example on outliers that made it easy to understand why you should consider removing them. Let’s say you want to try and predict the house prices in a specific neighbourhood. Out of 100 houses in the neighbourhood, 98 of them have 3 bedrooms, 4 bathrooms, and are around 1500 square feet. Two of the houses; however, are mansions. They are million-dollar homes.  

Now, those two houses are obviously going to skew the data and influence many indictors, like the median price or median square footage of a house in that neighbourhood. The two million-dollar homes are a good example of data you might want to remove or throw away completely.

Consistency is Key 

Olena explained how important it is to ensure the format of your data is consistent across the board. For example, if you are working with dates, you want to make sure they are written the same (MM/DD/YYYY, perhaps). Or if you’re working with phone numbers you need to determine if you want the “+” before the number or not. Either way, it doesn’t matter as long as it’s all the same.  

She explained that inconsistency is an all-too-common scenario with data and records. This will affect the quality of your data. As they say in the data world, “Garbage in. Garbage out.” If you are not using quality data, you will not be able to use it to make effective business decisions.  

Consistency matters. More than people realize. Inconsistency leads to mistakes which could cost an organization millions of dollars. 

Data Fairness

A couple of weeks ago, we published a blog that gave an example of data bias among school-aged children in which children of colour and children with disabilities were negatively affected.  

You might be using indicators that will be discriminatory or biased towards a certain individual or group of individuals. It is your job to ensure there is no discrimination, whatsoever in the data you are using.  

Access to Data 

Olena highlighted the importance of only allowing access to those who absolutely need it. There are always going to be people who argue that they need full access to data, but if you allow unbounded access to data, you are putting your business in jeopardy.  

You run the risk of data leaks and oversharing by granting access to whoever wants it.  

Conclusion 

We wrote this blog to highlight the importance of using clean data when making business decisions. While it is not a comprehensive, all-encompassing list, we hope it helped you understand why using clean it’s crucial.  

Always remember, folks: Garbage in…garbage out.  

Thanks for reading! Make sure to subscribe to our blog. We publish technology tips, tricks, and updates every week.     

Discover More