Validate and verify to ensure customer data quality

What is the use of customer contact data if you cannot be sure of its accuracy? Data that is not current, carries errors or is duplicated places a heavy load on the success of any organisation's subsequent initiatives. The building of a reliable contact database in an organisation forms the foundation of effective business intelligence and customer relationship marketing (CRM).

"Standardise, validate and verify data - and in that order," says Ardagh. "Data needs to be validated to conform to a standard. Only once that is achieved can the data be matched to outside records or external database to verify its accuracy. Verification of data is not sufficient on its own to create a data quality standard. Data needs to be validated and verified to achieve accuracy."

In South Africa, however, there is no specific standard against which to benchmark. The South African Post Office (SAPO) has drawn up postal delivery tables and issues the postal address management service supplier (PAMMS) certificate to companies with a 98% clean address database. Municipal tables can also be useful, using deeds information and service records to provide a basic standard for address information, although there are often discrepancies between these and the SAPO delivery tables.

Some large South African companies have established their own standard due to the lack of a national standard, while others rely on the contact information in a closed user group. Another popular method of updating contact data is using information from credit bureaux. If a name occurs frequently, it is considered verified. But this is too vague to be reliable, says Ardagh - while the information can be useful for a functional update process, it is not the core focus of credit bureaux to focus on address quality and they are unable to provide reliable detail to ensure data standardisation or validation.

"It's the greatest myth of data quality. Verification of data does not necessarily create a data quality standard. Verification on its own is not a validation process - data accuracy requires both verification and validation," says Ardagh.

Companies often turn to data cleansing and editing software to solve data quality problems. But, as Ardagh points out, not all data quality solutions are created equal.

Data cleansing software is designed to fix data quality problems according to a set standard. Two types of data quality errors occur when shifting data to a standard. Type One errors follow the "conservative" route - the data cleansing program passes over bad data. Certain errors are overlooked, the data passes all hygiene tests and it is considered "clean".

"Often this is because the error looks reasonable; other times it's simply because the data cleansing software is not up to the task, or the standard is inconsistent with the cleansing requirement within the company," says Ardagh.

Type Two errors follow the "renegade" route, where changes are made to data that initially had no errors. This forces new errors and create complex database problems.

"To avoid Type Two errors, the program must be set so as not to change anything that is already correct. Such caution includes the likelihood of missing something, or of committing a Type One error. Type Two errors are avoided by being so thorough that nothing is missed - and running the risk of over-cleansing," explains Ardagh.

Some data cleansing programs strike the balance between caution and being over-thorough by creating voluminous suspect reports. These programs catch everything suspicious, good and bad, and simply issue a report.

"This severely escalates the cost of achieving accurate data because the personnel and time cost greatly outweighs the software acquisition cost," says Ardagh.

Others compensate for Type Two errors by fixing very little. The poorest and most dangerous data cleansing program overcompensates for Type One errors and end up with data that is worse than it was at the beginning.

"A good data cleansing program is one that leaves good data alone, corrects bad data and creates short suspect reports with a minimum of set-up, maintenance and administrative review. It strikes a balance between Type One and Two errors," says Ardagh.

A reliable data cleansing solution draws on a sizeable knowledge base built according to a well researched and frequently updated standard that validates and verifies contact data regularly.