Although the opportunities to apply analytics in the public sector are abundant, cultural and technical challenges must be overcome before government agencies can claim to be fully developed, enterprise-wide, analytically competitive organizations. Building an analytical culture where data is widely used to evaluate hypotheses is crucial for an analytically competitive organization.
Despite the successes that the public sector has seen in the past with analytics, data analysis is not integrated into most decision-making processes. This can partly be attributed to the enormous variety of tasks in many different fields that government organizations perform. In such varied environments, one-size-fits-all approach to cultural change is often ineffective, and customized approaches training, policies, and incentives are necessary. These possible solutions require time and effort.
In both the public and private sectors, a scarcity of analytical talent makes it difficult for an analytical culture to flourish. Furthermore, unlike the private sector, many components of the public sector cannot outsource analytical work to contracting companies due to security and privacy issues. While competing to hire talent in public sector is difficult, it is not impossible; disadvantages in compensation (compared to financial and other quantitative industries that recruit analytics professionals) can be balanced by focusing on the importance of the mission: meaningful projects around public services.
The public sector (for example, the national security agency) manages a diverse and fast-changing set of risks and challenges. Terrorism, climate change, inequalities in natural resources, cybercrime, and the decline of the critical national infrastructure are just some of the challenges the agencies are trying to overcome to ensure the safety and prosperity of their countries. Meeting these challenges while juggling budget cuts requires a cohesive preventative approach across agencies like defense, police, intelligence, and border agencies. To succeed, a step change is needed in the way public sector organizations manage, share, and exploit their information.
The London Fire Brigade (LFB) has adopted a unique approach to integrate external data with its own and use predictive analytics to optimize performance. Internal records of where fires have occurred in the past are now integrated with a range of external data – including census information, land use, and lifestyles – to produce future fire risk maps of London. The maps inform budget allocation and enable the LFB to target its limited firefighting and preventative resources to the buildings and people most at risk. Put simply, using predictive analytics literally avoids the need for expensive and high-risk firefighting across the public sector.
Analytics in the public sector must also be responsive to unequal stakeholder groups. It is necessary, but not sufficient, to design analysis to satisfy the primary customer (often a decision-maker higher up in the chain of command); the wishes of the Parliament and the public must be considered, and they generally do not have a consensus on their objectives. When analytical findings are observed to favor one policy alternative over another, stakeholders may also challenge the analysis process to protect their positions and beliefs.
Outside the federal government, state and local governments are also working on these challenges, with varying levels of progress. Many of these organizations need the critical mass of analytical professionals to integrate analysis into all decision-making, and only a few have enough analytical talent to make substantial progress. At the local level, some innovative approaches have started, such as the New York City Police Department’s CompStat program, a data-driven view of police action that has extended to several major cities and has been extended with additional service functions. In the future, the “laboratories of democracy” that form an integral part of the American governance system will continue to develop new ways of using and approaches to analysis.
Finally, measuring the impact of analytics in the public sector is difficult because government success statistics are far more complex than simpler measures from the private sector. For example, counterterrorism decisions aim to minimize the number of fatalities, injuries, and economic consequences of an attack, within the limits of cost and impact on legitimate trade. Reasonable people agree that all these goals are important, but they do not agree on the values of the trade-offs between the objectives, so it is impractical to construct a single utility function to inform these decisions.
Efforts at all levels of the public sector give hope that analytics will be better integrated into decision-making, despite the challenges described above. The growing number of experienced analysts, the increasing availability of analytical tools that are easy to use and provide quick, easy-to-understand results; the accumulation of data and the dissemination of success stories all point to better integration. With many public-sector agencies giving priority to the development of analytical capabilities, analytics in the public sector will continue to grow, but this growth will require a constant effort to meet the challenges posed along the way.
For more on this topic, see Analytics In The Public Sector.