Wednesday, 20 January 2010

Questions to measure BI DQ/DM success

Encouraging and reacting to feedback from the business is a great way to measure the success rate of Data Quality & Data Management efforts within your organisation.

The audience to target for feedback should be business users/analysts - those people that consume data to support business decisions on a daily basis.

There are a number of ways you could conduct your feedback gathering:

  1. individual interviews could be held with each selected party
  2. a group session could be held, perhaps a free 'brown bag lunch' to encourage attendance
  3. a web-based survey published via the company intranet
  4. similar to the last point, but an e-mail survey

From experience, the most effective methods I have utilised have been the group session - as not many people can resist the idea of a free lunch - and secondly the web-based survey. Individual interviews often didn't materialise due to moving of calenders and BAU activities taking priority, and E-Mail surveys were often flagged and forgotten.

High-Level questions to consider

1. Are you happy with the quality of the data that you use for reporting?

The leading questions should ascertain to understand the general opinion on the quality of data that is used for reporting purposes. The aim here is to allow us to benchmark business opinion against quality dimensions such as accuracy, consistency, timeliness etc. Is data easy to use? Is it presented in such a way that allows simple understanding and interpretation?

2. Do you understand the meaning of all business terms and definitions relevant to you?

If there is confusion around business definitions, we may have data quality issues. If a business user suggests that their definition is different than a 'standardised' definition it may raise the issue for review of the definition and any reports that contain the term. Does the same definition differ between users? If so, who is using the correct definition for reporting purposes?

Likewise, if the answer to this question is a resounding 'yes', it may suggest that all the hard work that went into driving a single business glossary, and publishing it centrally for the business to reference and understand, has been successful (well done!).

3. Do you think the business terms and definitions you use are consistent in meaning across the organisation?

However, if the answer to the last question was a resounding 'yes', and the answer to this question is a resounding 'no', it's clear that we have a problem. Ask for examples of inconsistencies. This suggests that a revisit to business definition workshops, where the right people are in the same room to come up with agreed definition for terms, and which data sources/reports these apply to, is required.

4. Who do you rely on for data provision?

Do the business solely use known data sources to provide them with the data they need? i.e. the data warehouse. Or, do they also receive data from other sources (internal or external)?

This is a great question to aid in the task of mapping out the complete information landscape of an organisation. Understanding where data comes from, which systems it goes through, and which business rules have been applied at which point in it's journey is critical to improving and maintaining data quality and governance.

It will also help to understand whether, over time, there has become an increasing dependancy on 'information silos' to provide data to the business. Information Silos are often under the desk affairs created by analysts utilising data from the data warehouse while undertaking further manipulation to the data before passing it on to interested parties.

Why do these silos occur? Perhaps because the data warehouse doesn't provide all the required information, or doesn't provide it in a format fit for the consumption of a business user. Only by monitoring information usage across the organisation will we be able to ensure standardised data governance principles are applied, and look to reduce this silo culture.

5. Who do you supply data or reports to?

Likewise, if business users supply data or reports to other parties (internal or external), we want to know about it.

Why are they supplying data? Is there a reason this data cannot be extracted from source by the recipient? If it's a report, why can't the recipient self-serve? Is it an education issue, or do they not have the correct tools? Or, is source data not fit for purpose, so data is being manipulated prior to being supplied?

If data is being supplied externally does it comply with legislation i.e. data protection act. Is the data sensitive? If so, have data security and encryption methods been taken into consideration?

Only by having knowledge of all supply/demand avenues can we ensure the quality and control of that data.

6. Are you aware of all the data related legislation relevant to your role?

Compliance is playing an increasing role in all business sectors, and it is important that the education exists for business users to understand how data related legislation impacts upon their role. In the case of Data Protection - are they complying to DP legislation when reporting on data? What do they need to do to ensure SOX compliancy?

In Conclusion

Measuring success of BI DQ/DM activities by speaking to business users is critical in order to gain an in depth understanding of the practical issues faced on a day to day basis.

The key activity is to translate and publish the results, laying out a roadmap for how negative feedback will be actioned and resolved. Generally, people like to see the results of what they have contributed to. As well as increasing awareness towards DQ/DM matters, it also helps to increase the likeliness of future participation & cooperation.

Perhaps you already publish a Data Quality Dashboard within your organisation, looking at the technical side of how data is performing against set quality standards & expectations.

However, the Chartered Management Institute suggests:

"You need multiple perspectives of performance to manage your business successfully"

I consider the same thing to apply to DQ/DM activities, so perhaps consider something similar to the balanced scorecard approach in order to get a rounded picture of performance. I will touch on my experiences with this approach in a future blog post.

Until then, go speak to the business.


Post a Comment