- Kable, Sunday 4 October 2009 16.04 BST
Some local authority chief executives must have cringed on reading the recent Audit Commission report Is there something I should know? Its central warning, that many councils are basing important decisions on unreliable information, would have touched nerves and renewed worries that they are not doing enough to ensure their information is fit for purpose.
The report says that nine out of 10 councils acknowledge the problem, but that less than 5% have excellent quality data, while 65% face problems in sharing it externally. 80% of councils lack in-depth analysis, with analysts spending twice as much time on routine reports as anything that adds value. Two-thirds say their members struggle to understand information, and half say the same for senior officers. Also, half do not provide any formal training in the field.
It prompted Audit Commission chief executive Steve Bundred to issue a warning: "It's important that councils have the right information now because many of the decisions they are going to make in the years ahead will be harder than usual. Information may not sound like an exciting topic, but right now too many councils risk making poor decisions because their information is inaccurate, irrelevant or incomplete."
While the report does not make any recommendations on the use of technology, it raises a question about how far IT can go in solving the problem. Business intelligence systems have been designed to help organisations get the most out of their data, and the industry could see this as an opportunity.
David Waltho, head of government affairs for business analytics company SAS, says the role of technology is a significant omission from the report.
"For understandable reasons the Audit Commission chose to focus initially on the need for improvements in culture, people and process, because addressing these issues is essential to success," he says. "Nevertheless councils also need to understand that improvements only in these areas will not be sufficient. They will continue to struggle to control and make sense of the exponential growth in structured and unstructured data while that data resides in a multitude of unrelated databases, or they are reliant on resource-hungry and highly risky 'spreadsheet proliferation'.
"Information mature industries and the more successful councils have already shifted their IT focus from technology to information and have adopted integrated business intelligence and analytics to automate many of the data capture, cleansing, merging, analysis and dissemination tasks."
Waltho also points to the growth of predictive analytics in business, moving beyond the emphasis on identifying past trends to supporting a more proactive "predict and prevent" approach. This could provide a major asset for the public sector.
"Many public sector policies now put the emphasis on prevention, because it is both more efficient and effective to intervene before rather than after the event, but most public sector organisations have yet to acquire the tools for the job," he says.
He acknowledges, however, that although there is evidence that this helps to improve data quality and add business value, business intelligence has not been high on the priority list for many public sector chief information officers.
Michael Larner, a senior analyst in Kable's market intelligence team, says there are possible implications for the IT market, but suggests that, given the current financial outlook, few councils will be ready to spend the money.
"The report paints a pretty sorry picture that implies that, despite investing heavily in IT systems, many councils have suffered from a lack of understanding or complacency in making intelligent use of the data they collect," he says.
"This could be used in support of increased investment in business intelligence systems. The ability to collate data, mine it, gain a deeper understanding of its relevance and carry out predictive analysis is something any medium to large organisation cannot ignore, and the business case for the appropriate software is stronger than ever.
"But a lot of councils have already invested in such systems, and others have spent heavily on the customer relationship management systems that can provide some of the capabilities. The Audit Commission report, while it does not dwell on technology, implies that they have not made good use of these investments, and that chief executives should be demanding much better value for money from existing IT assets."
There is a view that the most important aspect of the issue is connected with good practice and the motives of people who are entering data, rather than technology. It is also something that is relevant throughout the public sector, not just in local authorities.
Philip Virgo, secretary of the parliamentary information group Eurim, says: "There are a lot of products for mashing data, massaging and supposedly automated error correction, but data is only correct if it is entered by people with an interest in, and the authority and knowledge, to ensure it is correct."
He suggests that, while computerisation in government has taken off over the past couple of decades, less attention has been paid to the quality of data management than the intricacies of the technology. There is now a need to reassert the discipline, with the sheer scale of errors on some databases posing serious threats.
"Audit Commission data shows a tremendous variation between different NHS trusts and police authorities in their audits. The proportion of records with errors range from 15% to 85%, and some of the errors may be terminal."
He says the electoral register provides a sharp illustration of the problem. When it is compared with data from the credit reporting agencies Experian and Equifax, it suggests that for some local authorities half of the people on the register do not live in the area while half who do are not on the register.
This contrasts with the Driver and Vehicle Licensing Agency's system for renewing car tax online, which has won widespread approval. Virgo says it has been successful because data comes from the insurance industry, whose members have an interest in keeping the data accurate and up to date.
"A lot of public sector information is not fit for purpose," he says. "A lot of the data is systematically skewed, because it is entered by people who know it will be used for resource allocation or performance management, and are therefore deliberately skewing it."
One of Eurim's workstreams is concentrating on the link between governance and the value of information, and ensuring that public databases are fit for their purpose.
"The big issue is whether your database is a valuable tool or a toxic liability," Virgo says, adding that audits of some health trusts and police records have shown serious rates of error. "If you were to say you were 'liable for liable' if you pass the information to third parties, the damages could run into the £100,000s."
David Waltho also emphasises the importance of following good guidance, pointing to publications from the Cabinet Office, the Treasury and the Knowledge Council, but suggests that part of the problem is that is widely spread.
"Although each report has much to commend it in isolation, few cross reference with each other and it can be difficult to connect the dots," he says. "Even assuming the target audience spot each of these publications, they may find it analogous to being sent individual pieces of a jigsaw and being asked to connect them without having sight of the picture on the box.
"There would appear to be a need for greater coordination and knowledge management across the public sector information management community."
Maybe the next step is for somebody to join all these dots.
