Data Quality – Breaking it Down

When you think of the phrase ‘data quality,' what does it mean to you?  Here are some questions I’ve heard users ask…

·         Is my data complete?

·         Does my data align with enterprise standards?

·         Why do I have missing data?

·         Why are there so many duplicates?

·         Why is my data giving conflicting information?

·         Why am I getting my data so late?

·         Is my data accurate?

The bottom line is that the questions are all over the board.  Why?  Because ‘data quality’ means different things to different people, usually dependent on when and how they interact with data.  Think about some of the reasons people might ask these questions and what their role might be…

Complete – A loan officer needs to know that the AI model uses a minimum set of required data to make application and product offering decisions.

Standards - A data analyst needs to know that fields named ‘CustID’ and ‘Col001’ are the same unique identifiers to join two tables together.

Missing – A marketing campaign manager needs to build an email marketing list to current customers.

Duplicates – A business manager needs to know that customers are only being counted one time.

Conflicting – A data scientist needs to know that their analysis is not going to be challenged by the ‘same’ data that is available in an operational report.

Late – A supervisor needs to make production staffing decisions based on current orders.

Accurate – A financial analyst needs to know that the data they are using on a reporting platform can tie back to the source system.

I know these are simple examples.  I know this is not as exciting as AI.  However, if you can’t trust the data you are using on any platform (including AI models), you will find yourself in a world of hurt. It is some of the simplest ‘data quality’ issues that take users an extraordinary amount of time to ‘fix’ or create temporary work arounds. 

I’ll break down thinking about the different pieces of data quality next time.