Data Quality Before Data Analytics

A sophisticated analysis cannot repair a population that was defined, extracted or transformed incorrectly. Before interpreting a pattern, establish whether the underlying data can support the question.

Start with the decision and the population

For an illustrative duplicate-payment screen, decide what counts as a payment: a bank transaction, a payment document or an invoice allocation. Those units can produce different row counts and different matches.

Specify entities, systems, period and transaction status. Understand how instalments, reversals and credit notes should be represented. An export of approved invoices may be relevant to one question while omitting the actual payment event needed for another.

Reconcile what was extracted

Record the extraction time, filters, source report and responsible owner. Compare record counts and control totals with the agreed source population, separating currencies and transaction types where needed.

Investigate differences rather than forcing the totals to agree. A reconciliation can help identify omissions or extraction problems, but an agreed total does not prove that every record is accurate. Assess source reliability and corroborate critical fields against suitable supporting records.

For financial-report audits under Australian standards, ASA 500, paragraph 9 addresses the reliability of entity-produced information, including accuracy, completeness and sufficient detail for the auditor's purpose.

Validate fields without confusing format with truth

Check mandatory identifiers, data types, date interpretation, currency and amount signs. Preserve leading zeros where they form part of an identifier. For timed events, document time zones and cut-off rules.

A date can satisfy a format rule while referring to the wrong event. The Government Data Quality Framework distinguishes validity from accuracy and treats quality as fitness for the intended use.

Check the effect of joins and transformations

Linking a payment file to invoice or supplier data can drop unmatched records or multiply rows through a one-to-many relationship. Compare counts and totals before and after the join, and explain the expected relationship.

Keep the original extract and a reproducible transformation log. Document normalisation rules; a change to invoice identifiers may create a match that did not exist in the raw data.

Preserve exceptions for investigation

Do not remove repeated payments merely to make a file look cleaner. They may be the very condition being investigated. At the same time, similar supplier, amount and invoice fields do not establish an improper duplicate payment.

Retain flagged records, investigate the underlying events and distinguish a repeated representation of one transaction from two separate payment events. Document justified exclusions so another reviewer can reproduce the analysis.

Use a practical quality gate

• Define the population: align scope and record unit with the question.
• Reconcile the extract: investigate counts and control-total differences.
• Validate critical fields: distinguish valid format from accurate information.
• Check transformations: account for joined, dropped and multiplied rows.
• Preserve exceptions: retain anomalies and explain exclusions.
• Record limitations: state what the data can and cannot support.

These six steps are an original practical structure, not a mandatory checklist or the framework's six quality dimensions. The UK framework is useful input-quality guidance; analytical logic and interpretation still require their own review.

Quality checks can be embedded in the analysis and revisited when new issues emerge. ASA 500, A61 recognises that evidence about accuracy and completeness may be obtained concurrently with the audit procedure when integral to it. Assess reliability before relying on the conclusion.

Report a supported conclusion

If a critical field or part of the population remains unreliable, obtain better information, narrow the conclusion or pause the affected analysis. Avoid presenting "no matches found" as assurance when the data could not reveal the condition.

A quality gate does not require perfect data for every purpose. It requires a transparent decision about whether the remaining limitations are acceptable for this purpose.

Before trusting the pattern, understand the population.