Small Data, Big Insights in Health

Big insights can be drawn from small data

When the state of Florida announced the result of that state’s vote in the 2012 US Presidential Election, it confirmed that the election had produced not one, but two big winners.

In addition to Barack Obama, the economist and statistician Nate Silver had a triumphant performance. He visibly and very publically forecast accurately the results of all 50 states plus the District of Columbia. Immediately before the election, Silver published a typically iconoclastic book on the uses and abuses of data in general, and statistics in particular. Big Data does not escape the lash of Silver’s pen. He states that “the volume of information is increasing exponentially, but relatively little of this information is useful – the signal to noise ration may be waning. We need better ways of distinguishing between the two.”

This assessment may seem like a dash of cold water on the hopes for Big Data in Healthcare, which the Wall Street Journal recently tagged as one of the biggest potential areas of application for society. It is easy to see from whence this hope arises, as the accumulation and use of data in healthcare has historically been very difficult. The combination of clinical autonomy, transactional focus, regulation and stringent safeguards has led to a dual-speed approach to the use of information. Individual doctor-patient interactions tend to make use mainly of the information available to the doctor at the time, while changes to treatment protocols are subject to the highest standards of evidence. In this environment, the data needed for planning, management and performance is often an afterthought, emerging unnaturally from treatment systems, and forced to comply with inappropriately high standards of evidence. For these and other reasons, I am certain that the WSJ’s assessment is correct in at least one regard – there is enormous scope to improve healthcare delivery through the judicious application of data.

This is an extract of the article. To download a full copy of the article, please subscribe to our mailing list.