How Can Data-Driven Decision-Making Go Wrong? What Can We Learn?
Effectively using data for decision-making in organisations can be challenging in practice. In this post, I explore three ways in which data-driven decision-making can go wrong and what we can learn from them.
1. Basing decisions on limited data and metrics without considering the adverse consequences
Organisations can sometimes rush into using whatever data is available to make decisions, without fully considering the potential negative consequences.
Example: In the education sector, as attention shifted towards improving the quality of schooling, metrics such as the pupil-class ratio and the pupil-teacher ratio were readily available, monitored, and often used for target-setting. While these metrics are useful for understanding some aspects of schooling conditions, they do not directly measure learning outcomes (for example, see here and here). Also, focusing too heavily on these metrics may create adverse incentives, such as encouraging the rapid construction of low-quality classrooms or developing shorter, lower-quality teacher training programmes.
Lesson: While using the available data can boost data familiarity and literacy, laying the groundwork for a data-driven culture, organisations should first invest in: (i) developing metrics that track ‘real’ progress (for example, learning outcomes); and (ii) tracking metrics that signal potential adverse consequences (for example, on the quality of classroom conditions and teaching). Without striking this balance, data-driven decision-making may lead to suboptimal or even harmful choices.
2. Expanding data collection without working backwards from key decisions
When organisations begin collecting data in new areas, there can be a tendency to gather as much data as possible, as quickly as possible, and across as many variables as possible.
Example: The recent expansion in data collection on the public-sector workforce has led to new insights into the characteristics of employees and managers in the sector (for example, see here and here). However, when trying to use this data to set strategic direction (for example, to answer "what drives public-sector performance and productivity?"), key variables can often be found missing (for example, on performance measures or key variables for causal analyses).
Lesson: New data-collection investments should start with a clear understanding of the decisions and analyses they will support. By explicitly listing the individual questions and ideal variables for each in advance, organisations can ensure more focused and useful data-collection investments, leading to better data-driven decision-making.
3. Not sufficiently investing in causal inference methods
To inform high-stakes decisions, leaders often ask analytics teams to look for changes in the trends of key metrics without sufficiently investing in efforts to rule out potential confounding factors. Without understanding the isolated impact of a new investment, it’s impossible to reach an accurate measure of its return.
Example: When launching a new membership programme or feature, leaders often focus on the number of sign-ups or customer adoption rates over time. They might label the new launch a success if there is a certain percentage increase in the metric since launch (for example, if the number of sign-ups increases by 5% since the launch date). However, leaders tend to put less emphasis on isolating the impact of the new programme or feature from other potential drivers, such as seasonal trends, organic growth rates, or launches by other teams — especially if the result is favourable to their new launch.
Lesson: Causal inference methods should be used in parallel with traditional data analytics, especially for high-stakes decisions. Laying the foundation for such methods, such as staggering the rollout of a new feature or programme, maintaining a 'holdout' group, or making use of the discrete thresholds in classification models, can help determine the true (isolated) value of an investment or launch. Qualitative evidence from stakeholders can also provide valuable insights into a programme’s success and its potential drivers. Planning ahead and considering such methods in advance of new investments can substantially reduce their implementation costs and improve the quality of the insights that they produce.