In Data Quality

Why you need to complete and standardize the information going into your database

Your business data is more than just bits and bytes lodged in a server farm somewhere. Business data goes beyond just customer data and the hardware and software that store it.

Business data is a product of the process of collecting, standardizing and distributing that information to those who will use it. If the collection process is flawed the data will be flawed. Quality data is the foundation of any database, whether it’s a project specific database someone whipped up on Access or a multi-million dollar information system. The value of the data is in direct proportion to the quality of the data.

Factors That Impact Data Quality

Poor data quality is caused by more than just fumble-fingered typists.  While data entry errors do cause problems, the usefulness of a database can be even more limited by its structure, quirks in how the fields are set up and lack of standardization of common fields. These problems can become even more pronounced when trying to migrate data from one database and merge it with another. Factors that impact data quality include:

1. Obtaining bad data from supposedly reliable sources – Too often information systems managers trust sources of information they shouldn’t. It’s best to take a hard look at any data before you bring it into your system.

2. Faulty data collection processes – Data collection processes that use faulty collection instruments collect faulty data.  These instrument flaws can include fields that record ordinary information in non-standard ways such as collecting birthdates in memo fields. Data collection instruments that ask ambiguous questions tend to collect ambiguous answers. Multiple choice questions can leave out critical choices. Required fields may ask for information many responders would rather not provide. Those who choose not to answer may find themselves unable to finish surveys without providing the objectionable information and abandon the survey. This sorts an entire market segment out of the sample and can make the sample invalid.

3. Miscommunication – If data collectors misunderstand what the survey instrument expects good data to look like, they may alter the data to fit the format of the question or make up data to fill an empty field instead of leaving it blank.

4. Failure to Maintain and Update the Data – Data and especially demographic data tends to change over time. Failure to keep up with the changes in contact information, age and to review the data, remove duplicates and damaged records results in inevitable deterioration of the data your company depends on.

5. Misusing Otherwise Good Data – Calculation and processing instructions, filters and formats applied to databases can alter or misdirect data and spoil its effectiveness for use in decision-making and planning.

6. Trust in Automated Systems – As users come to trust automated information systems, it becomes ever more critical that information systems start out with high quality data – “Garbage in, garbage out!” as the saying goes.

Data Quality Improvement

The only way to insure you are collecting high quality data is to create a data quality improvement program.  The project includes representatives from all departments and levels of the company. The team evaluates your company’s need for data, the actual quality of its existing data and follows a step by step development process that engages all the business systems of the company. The data quality improvement process is designed to insure the following:

●     Completeness of data collection

●     Validity of data collected

●     Structural integrity of the database itself

●     Business-Rule Integrity

●     Standardization of data in conformance to conventional database standards.

Information Systems Responsibilities

 

To reach a consistently higher data quality standard, the information systems staff must prioritize sources from which information is gathered, drawing from highest quality sources first. The IS team should be the cheerleaders for implementing organization wide data quality policies and take the lead in promoting standardization of data fields used by all departments.

Standardized data fields insure that everyone is collecting the same types of data wherever they collect it. Identify your company’s critical data quality needs. To do this policies and procedures must be created that promote accurate data collection no matter what department or system within the company generates the data.

A data quality improvement project will:

  1. Facilitate information exchange
  2. Reduce data maintenance costs
  3. Improve everyone’s access to data
  4. Reduce the cost of data maintenance
  5. Provide richer levels of information
  6. Allow easier access to information throughout the organization
  7. Improve communication and coordination between systems and departments