Designing new public institutions for the UK in the 2020s and beyond

Ahead of the general election, attention is beginning to turn to what new institutions might be created by a new government, with Labour promising a Great British Energy, a National Care Service, Skills England and an Office for Value for Money. 

The paper (which can be read here, or in the link below) looks at the history of institutional design; why some institutions survive, and others don’t; lessons to be learned from other sectors and other countries; some new design principles; and the many potential priorities for creating new institutions in the UK over the next decade. 

It has been written ahead of a session I am co-hosting with Nick Pearce at the British Academy in May 2024 and links to a global programme of work being done by the Institutional Architecture Lab (tial.org) with various national governments, the Rockefeller Foundation and the UNDP.

Designing new public institutions for the UK in the 2020s and beyond

.pdf

Download PDF • 271KB

Background

The UK has a great history of creating public institutions – from the Post Office and BBC to the NHS, the Metropolitan Board of Works to the British Library, the Arts Council and the Open University.  Some were very much part of government, while others were designed to serve the public interest while remaining independent.  Some became part of the furniture of everyday life, while others soon disappeared.

Winston Churchill famously commented that we shape our buildings, but they then shape us. The same is true of institutional architecture – we shape our schools, courts, hospitals, regulators and parliaments, but they then shape not just what we do, but also how we think.

Institutions exist because distinct tasks - running libraries, hospitals, armies, supporting science or providing welfare services – require a distinct ethos, methods and capabilities rather than generic bureaucracies.  It follows that new tasks will often require new institutions too – and as I show there is a lot of activity globally around tasks ranging from the governance of AI to decarbonisation and mental health.

Perhaps surprisingly, however, there are no centres of expertise in institutional architecture, either in the UK or globally, and political parties, and governments tend to default setting up committees or commissioning consultancies to help with design, which often leads to a recycling of traditional models rather than fresh thinking about what might work best.   

Things look very different in the world of business, which is served by hundreds of centres interested in organisational innovation and feverish debate about new forms.  These have contributed to dramatic change in how organisations are structured, including vast global businesses centred around search engines (Google), algorithms (TikTok) and platforms (Amazon, Alibaba).  A generation ago, few imagined that the world’s biggest taxi companies (Uber and Didi) would own no taxis or that the world’s biggest provider of accommodation (Airbnb) would own no hotels.  By contrast, there has been relatively little creativity in the public sector where thinking about institutional design has not advanced much in recent decades.

New institutions define the character of governments – but pose dilemmas

New institutions played decisive roles in British history, often defining the character of governments, and often contributing to dramatically different outcomes for citizens.  The NHS, British Railways, New Towns, Coal Board and many others helped define the character of the Attlee government and its legacy.  The Thatcher government sought to define itself as efficient and business-oriented through the Next Steps Agencies, the privatisations, which then required a new family of regulators, and organisations like the London Docklands Development Corporation and equivalents in other big cities intended to bring a ‘can-do’ ethos to urban regeneration.  The Blair and Brown governments left a legacy of SureStarts, devolved administrations and elected Mayors.

Over the last few decades many new institutions have been created. They include dozens of local TECs, LEPs and Learning and Skills Councils, national institutions including LearnDirect, NOMS, Connexions, the Equality and Human Rights Commission, National Crime Agency, Office of Budget Responsibility, Infrastructure and Projects Authority, UK Space Agency, Office for Students, Financial Conduct Authority, Health Security Agency and the National Institute for Clinical and Health Excellence, UK Research and Innovation and the Advanced Research and Innovation Agency. Many have been created in the devolved nations, from the Scottish National Investment Bank to Wales’ Future Generations Commissioner and this year’s Independent Commission for Reconciliation and Information Recovery (ICRIR) in Northern Ireland. 

However, despite this flood of new institutions, ruling parties have often been sceptical about the role of new institutions. Some Conservative governments had little interest in public institution building for ideological reasons (their priority where possible was to pass functions out to the private sector), and the leaders of the last Labour government had much less interest in leaving behind legacy institutions than their predecessors.

There were some good reasons for this scepticism. Any government, especially one with limited resources, is bound to worry that misconceived new institutions can get a good headline for a day but then tie up lots of management and legislative time, and money, to little effect.   This scepticism is often encouraged by senior figures: the ‘iron law of incumbent scepticism’ states that people who have risen to the top of existing institutions will be genuinely sceptical of the need for new ones and invariably convinced that new tasks should be given to old institutions (and are sometimes right).

Another reason for scepticism is that the relationship between institutions and social and economic outcomes is complex.  Past theories attempted to show that growth led to the creation of robust institutions, or conversely that strong institutions fuelled economic growth, or, again, that nations were trapped by their history and had little freedom to change their institutional destiny.   All of the theories can explain some phenomena – but there are also so many exceptions that it’s wise to be cautious about over-generalisation.

Longevity and stickiness

A related dilemma concerns longevity: there is little point spending a lot of political capital on new institutions if they are simply abolished as soon as another party is in charge.  Many new institutions have indeed proved short-lived, including the TECs, LEPs, LSCs, Audit Commission, RDAs, Connexions, Health Promotion Agency, the UK Film Council and many others.   

Why have so many proven flimsy? There are many possible answers. Some were abolished for reasons of political symbolism. But perhaps too many had been set up to give the appearance of action rather than being based on sufficiently serious thought; perhaps too many used standard defaults, from NDPBs and Executive Agencies to regulators, that weren’t fit for purpose; and perhaps in too many cases the design focused solely on authority, finance and organograms, rather than issues of culture, intelligence or relationships.   

Since all new institutions face difficulties – setbacks, scandals and failures – a critical question is whether, when these happen, they are given the benefit of the doubt both by the public and by relevant stakeholder and expert communities.   Some of the institutions which have survived longest combined a strong moral ethos and sense of mission along with competence and excellence.  Some – notably BBC and NHS - had a direct relationship with the public, including very regular direct communication.  Others were more technocratic but won over key stakeholders and experts (NICE is a good example) or were lucky with their enemies (such as the OBR which was greatly strengthened by its brief interactions with the Truss Premiership).   A tentative conclusion might be that some mix of moral purpose and mission, perceived competence and strong relationships is key to survival.

What can we learn from other sectors and other countries?

History provides many insights into which kinds of institutions are likely to prove most successful, and clearly the options open to any country are strongly shaped by decisions taken in the past, by prevailing cultures and norms. Path dependence matters a lot for institutional design, even if severe crises, from wars to pandemics, can open up new options.   

But understanding of the past needs to be combined with understanding of the present and future since the organisational options in any era reflect prevailing values and the available techniques and technologies. 

Changing values have left many existing institutions struggling with the perception that they are hide-bound, rigid and hierarchical, forcing greater attention onto public engagement and transparency, as well as accountability for issues such as environmental impacts and equality. These shifting values have encouraged the creation of public-interest institutions partly or wholly independent from the state: mutuals (like Welsh Water); endowments (like Nesta or the Education Endowment Foundation); social mission finance organisations (like Big Society Capital); and many providers of care, housing and other services spun out of the public sector.

Technologies and techniques have in parallel changed the options available.  Since much of the everyday life of organisations concerns communication, data, knowledge and decision-making it’s not surprising that an era of dramatic innovation in these fields has also affected organisational design.  Most of the world’s biggest companies by capitalisation (from Meta and Alphabet to Bytedance/Tiktok and Amazon) are now structured around platforms, algorithms and data. There has also been plenty of innovation in civil society through everyday platforms like Wikipedia or models such as Buurtzoorg in care, and the wide array of mutuals, coops, B-Corps and CICs (of which the UK now has some 29,000).  Some of these benefit from much lower costs of coordination that have enabled ‘mesh’ structures that mix horizontal and vertical in ways very different from the classic 19th century silos that still persist in much of the public sector.

There has also been much invention and speculation around the use of blockchain, for example with over 13,000 blockchain-based DAOs (decentralised autonomous organisations) managing over $20bn in assets, and more recently interest in how best to integrate AI into organisational design, whether for managing everyday processes (from work allocations to negotiations) or supporting innovation.   These models tend to point towards flatter structures, more flexibility and a greater emphasis on how intelligence is organised. 

However, the design of public institutions in the UK has hardly been touched by these developments.  Few of the ones created in the last decades has used any of the technological ideas described above. 

There has also been surprisingly little learning from other countries.  UK government has had a difficult decade, thanks to austerity, Brexit, COVID and political instability.  However, this hasn’t led to a more outward-looking approach.  Instead, most recent institutional design has essentially relied on UK precedents (it’s also striking that nearly all the recent reports on the organisation of Whitehall and the public sector have relied for the great majority of their inputs, and reference points, on UK examples).  

Yet there are many lessons which can be learned from other countries, even if particular models are rarely replicable in precise ways.  Potentially relevant examples range from new welfare providers (like the National Disability Insurance Scheme in Australia, which now accounts for 2% of GDP) to new industrial policy vehicles (like the multi-trillion Government Guidance Funds in China), new ways of influencing labour markets, like South Africa’s Commission for Employment Equity (CEE) to new ways of organising care (such as Colombia’s Care Blocks). 

No-one could claim that the world’s governments have yet moved decisively into the Internet age (though the UK has done relatively well with gov.uk, simplifying many everyday interactions with citizens), let alone the age of AI.  But there are many striking examples where digital technologies have inspired radical new designs, including new ways of organising identity (such as India’s Unique Identity Authority, responsible for Aadhaar, and recent offshoots like the Open Networks for Digital Commerce); the digital linking structures like Estonia’s X-road and Ukraine’s Diia that support a wide range of public services, as well as business; or vTaiwan, the radical innovation in democracy linked to Taiwan’s parliament.

Other countries can also provide useful prompts for more imaginative thinking.   Denmark for example shows how national and local functions can be interwoven (local government plays a role in raising taxes), or the crucial role of funds and endowments (such as the National Building Fund - Landsbyggefonden – LBF, one of many reasons young people in Denmark are so much better provided for than their UK equivalents), and the recent creation of a Danish Nature Fund (2015) and a Social Investment Fund (2019).

There are also current projects underway that may have a lot to teach the UK.  Germany for example is creating a new agency - DATI - to promote technology adoption in business, the public sector and civil society – which is likely to be a central issue for addressing the UK’s productivity problems.

The ones mentioned above are intended to be permanent. But in many countries there has been a shift towards more time-limited and temporary structures. This has always happened to some extent during wars and crises but is now coming to seen as useful in other contexts. For example, the UAE has even done this for ministries (through its Ministry of Possibilities), and this approach is clearly relevant to problem solving and missions.  

Emerging design principles

Public organisations are, and should be, very diverse.  Past attempts to squeeze them into standardised forms, such as Executive Agencies, often backfire.

But there are some emerging design principles which can help to broaden the range and relevance of options in many different fields.   Some of these are described on the TIAL website which includes case studies as well as overviews of institutional architecture and toolkits for design.  We particularly highlight five principles which can help in the work of design:

·       Dynamic metaphors: different metaphors can guide design, so that instead of relying on traditional pyramids and classic organograms, alternatives can be drawn on including networks, flotillas, myceliums and other ideas taken both from nature and computer science rather than traditional public administration. 

·       Shared intelligence: in the past the primary focus for institutional design was law and economics, or to put it another way, the organisation of authority and finance. These remain important but intelligence, data and knowledge are now increasingly central to the design of institutions, as they are in business.  This includes not just what happens within institutions but also how they serve whole systems, ensuring that people, businesses and other organisations are supported with the intelligence they need to act successfully.

·       Meshes as well as silos: a related point is that governments are increasingly seeking ways of organising cross-cutting ‘meshes’ that allow ‘whole of government’ action, connecting multiple silos, multiple tiers, and multiple organisations, again often with shared knowledge.   

·       Sparring and ecosystems: institutions sit within a landscape or ecosystem – of cooperation, competition and challenge. No institution is an island.  So, the work of design has to consider what surrounds the new institution, including how it may best be challenged so as to avoid complacency and stagnation.

·       More voice and inclusion: finally, there is growing interest in new ways of organising voice and accountability – involving beneficiaries in shaping decisions and ‘co-producing’ services, and sometimes giving more formal power to communities, as well as tapping into much wider sources of expertise and collective intelligence.

The richer menu of options that flows from using these design principles may help to create more agile, effective and legitimate institutions in the period ahead.[i]

Some priorities for future institutional design in the UK

How might these ideas be used in the UK?  Over the next decade the UK looks likely to need many new institutions to fill a series of governance gaps – from regulating AI to orchestrating decarbonisation, providing care and mental health, tackling low productivity or ensuring cybersecurity.  

Labour has already promised to create ‘Great British Energy’, ‘Skills England’, an ‘Office for Value for Money’ and several others. Labour and the SNP have promised National Care Services; and the Conservatives promised to make the UK a leader in AI regulation through the AI Safety Institute.  

Here I briefly describe some of the likely fields for institutional innovation in the UK in the late 2020s, in all of which current arrangements are unlikely to prove adequate.

Coordinating institutions for skills and productivity - many public institutions were created in recent decades to coordinate the work of public organisations and business on skills, apprenticeships and economic development, including LEPS, TECs, LSCs, RDAs and others. None proved sustainable and investment in skills training and learning has declined. Skills policy is commonly divorced from discussion of job quality, work organisation, and firm level innovation, and reduced simply to supply of skills, rather than how they are utilised by employers. So, attention will need to turn to new options, such as how to integrate skills policies with strategies to increase innovation and raise productivity, give strategic direction to skills provision, from FE to apprenticeships, and foster regional economic development. Many countries are experimenting with new models, such as Jobs Skills Australia, or Singapore’s SkillsFuture, and this may be a field where some standardisation at a national level in relation to data, knowledge and navigation tools, as well as sectoral industrial strategies, can combine with devolution of operations and funding.

Partnership organisations around pay – many see an important role for public institutions to guide the evolution of pay and rewards, redressing glaring inequalities.  The Low Pay Commission is a rare recent success, linking government to business and successfully legitimating minimum wages.  Labour has committed to a ‘Fair Pay Agreement’ for social care for example, prompting the bigger question of whether the UK should copy other countries and return to strong sector pay bodies? 

Care – there has been much discussion in recent years about creating some form of national care service to mirror the National Health Service. But details are usually missing, and most attention focuses on finance.  Getting the fund right will be crucial, but so will the organisation of knowledge and data, values and ethos, oversight and support for performance, focused innovation and an active development role to fix the sector’s many problems.  Any new institution will need to be some kind of mesh, linking in with local government, and adopting the ideas of a more ‘relational state’ that have been discussed for two decades or more.  Yet care currently lacks many basics – with a huge amount of work to be done to raise the pay, security and skills of staff and address very low levels of digitisation. This probably points to a stepped approach to creating any new national framework or body, but options are very overdue.

Mental health – while mental health has steadily risen up the public and policy agenda, the imbalance between its institutions and those concerned with physical health has become ever more apparent.   These are more fragile, less visible, less well-funded and significantly lower status.  But what would work better?  This is both a question for national governments and for cities and regions, as they grapple not just with the most acute needs for treatment and care, but also with issues of population-level mental health.  The state of Victoria in Australia is an interesting current example that is following the recommendation of a Royal Commission and creating a new institution to both shape and deliver services, the ‘Victorian Collaborative Centre for Mental Health and Wellbeing’.

Housing and planning – housing is set to be one of the UK’s top priorities over the next decade, and there is an evident capability gap in terms of achieving significantly higher levels of housebuilding.    In one view the key is to give local authorities more powers, and funds to act quickly.    Another view sees the issues primarily through the lens of incentives, assuming that developers will play the central role.  But new institutions may be part of the answer, as they were in past periods of history, orchestrating resources, planning and design, and potentially ensuring better ways of recycling land value gains into new building.  At a minimum, the roles and powers of Homes England will need to be rethought.

Infrastructure the National Infrastructure Commission was set up in 2015.  Its role is to do reviews and make recommendations – but these often highlight the lack of institutional capacity for concerted action.   A new government will need to consider whether a much stronger agency is needed to plan and oversee implementation, ideally with cross-party buy-in.

National and local coordination - a crucial cluster of issues concerns how to align national and local action in terms of funding, standards and implementation, on topics including skills, transport, policing, care and more. Some of these responsibilities are being absorbed into Combined Authorities. But even in these cases some ‘mesh structures’ will be needed with some functions shaped at a national level, some centralised orchestration of data, knowledge, standards and evidence alongside local delivery and prioritisation.   There are some obvious institutional gaps: no UK city has a well-developed institution to connect multiple tiers of government to universities, so that they contribute to solving public problems (like the London Collaborative in the 2000s).  None has quite the partnership structure needed for decarbonisation (though some like Bristol with its City Leap partnership have promising approaches).   None has an effective way to link data in a systematic way to help solve problems, despite good initiatives like LOTI in London and Data Mill North in Leeds.

Institutions for the transition to a net zero economy   Every country is grappling with the design of new institutions of net zero economy which seek to embed mutual commitments from businesses, sectoral bodies, and governments at every level.   There are some impressive examples of collaborative structures overseeing pathways and roadmaps, and in the past the UK experimented with a range of institutional innovations around climate, including the 2008 Climate Change Act, with legally binding targets, a Department of Energy and Climate Change, a Climate Change Committee to monitor progress and the Green Investment Bank.  Some of these were later dismantled and the UK is struggling on decarbonisation (notably in relation to home energy and heat pumps) with a lack of institutional capacity sometimes cited as a factor.   Other countries have created many institutions to help, including sectoral pathways and roadmaps, and a plethora of coordinating bodies: the UK will need to decide what institutions and structures it needs at multiple levels.

Science and technology UK Governments have set up many institutions around science and technology in recent years, including UK Research and Innovation (UKRI) combining multiple funding agencies, and ARIA to finance higher risk technologies.  The jury is out on whether these will succeed, and one observation is that little serious analysis was done to learn from other countries experiences (for example, from Norway in relation to UKRI, or from France and Germany as they attempted parallels to ARIA).  My own view is that the UK should have created some version of ‘Advanced Systems Agencies’, linking R&D, policy and regulation in fields such as transport and energy, and holding deep knowledge of these fields, rather than the generic approach of UKRI and ARIA.  But further restructuring could be a diversion. Nevertheless, there are some important gaps – particularly around technology diffusion and adoption; handling the governance of key technologies, from quantum to synthetic biology; and the organisation of data and knowledge around research.   

Artificial intelligence As AI becomes ever more central to societies and economies attention is belatedly turning to new institutions to maximise opportunities and reduce risk.  Some institutions already exist. China’s Cyberspace Administration was set up in 2014 and has become increasingly involved in implementing new AI laws.  The UK set up a Centre for Data Ethics and Innovation within government to think through dilemmas of AI tools like facial recognition and targeted advertising in 2018. But it never won political engagement.  As the EU’s AI Act comes into force in 2025 member states are likely to have to create regulators.  Rishi Sunak has created an AI Safety Institute (though few expect it to survive in the current form) while continuing to argue that existing institutions for competition, consumer protection or personal data can handle everything, an approach mirrored in the US, where responsibility is split between existing organisations like the Federal Trade Commission and the National Institute of Standards and Technology (which focuses on standards for AI), alongside another new AI safety institute.   But many of the issues raised by AI are crosscutting – such as facial recognition or liability – and the history of past technologies suggests that it is unlikely that reliance on existing institutions will be enough.  The car is now governed through a lattice of institutions in most countries, whether responsible for vehicle licensing, highways management and construction, testing drivers, regulating safety or emissions, managing parking or handling taxation.  We should expect a comparable pattern for AI.

Civil service skills - the dismantling of the UK civil service college in the early 2010s is widely seen as a mistake (and has not been copied by any high performing governments).  But what might a replacement (perhaps His Majesty’s College of Government’) look like?  Should it aim to cover all sectors or just national government?  What curriculum would it need? Should there also be a focus on capability – as with proposals for ‘His Majesty’s Inspectorate of Government’?  A related question is training for politicians.  Some countries provide extensive support for skills, notably China and the US, and Australia recently saw the creation of the McKinnon Institute.  The UK has almost nothing.

Productivity - there has been some discussion recently of the creation of a UK equivalent to the Australian Productivity Commission, to provide a centre of expertise and policy advice. New Zealand is currently dismantling its equivalent; but what might such a body look like in the UK?  And is this best conceived as an advisory, high-level body or a more practical, action-oriented one?

The essentials of everyday life - what might be institutional expressions of the current interest in a ‘foundational economy’?   Most of the systems that shape everyday life – food, energy, money – are very opaque, and mainly beyond the scope of public institutions, though some are attempting to reshape local procurement and local welfare provision.  Are there plausible options that would make the ‘foundational economy’ more than a slogan and more amenable to democratic shaping?

Global proposals – the UK was in the past active in the creation of global institutions, from the UN to the IPCC and IPBES.   There are now innumerable governance gaps globally, from space and oceans to artificial intelligence and capital for energy transitions.[ii]  Should the UK have distinctive view on how to better fill these gaps and enable some limited growth of multilateral institutions?

Many others could be added to this list – from a new institute for ethics and integrity (there are well-developed plans for this in Australia as a response to the many instances of moral failings by public and private organisations); new institutions to drive value for money and efficiency; and from procurement to supporting public engagement through citizens assemblies and other tools.   There are also many opportunities to remake existing institutions and give them new remits – from Crown Estates (which already plays a big role in energy) to the Electoral Commission (which may need significant new powers to protect elections from disruption).

The 2010 government drew up a long list of institutions to abolish.  The next government will need a list of which ones to create, while also hopefully culling some which are no longer fit for purpose.

The human dimension

Finally, it’s important to recognise the human dimension of institutional design since institutions that work well on paper may crash in practice.   One aspect is leadership – early leaders often define a new institution, though the ideal leader in a start-up phase may not be ideal in a phase of consolidation.  Another aspect is the role of informal networks and structures.  A consistent message from research is that informal structures and networks are as important as formal structures in helping institutions succeed, including relationships of trust with key stakeholders.  This can be a blind spot in Whitehall which usually focuses on the organograms rather than the relationships around them.   Then of course there is the morale and ethos of the people who work in an institution – again, where there is a strong sense of mission or moral purpose it’s bound to be easier to ride out crises.   And finally, as indicated earlier, new institutions can benefit from their own direct relationship with the public, who will then fight for their survival, or, in the case of more specialist bodies, it may be vital to maintain the respect of experts and communities of practice.

I hope that this paper will stimulate both thought and debate about priorities for the next decade.  The winner of the next election will face extremely severe fiscal constraints, as well as limited bandwidth.   As a result, not much can be done quickly or easily.  But the same was true in the 1940s.  The difference is that then a lot of hard work had been done to think about the UK’s future institutional architecture.  Equivalent work is seriously overdue now for the late 2020s and 2030s.

………………………………………………………………………………………………………………………………………………

[The Institutional Architecture Lab, TIAL (tial.org), has recently been created to open up a more expansive debate about how best to design institutions for new tasks, drawing on the lessons of history while also making the most of contemporary tools.  With backing from the Rockefeller Foundation and others, it’s working with various national governments, cities and the UNDP to prompt more imagination, focused on practical live cases while also working on theory and models.  This paper applies some of its thinking to the UK context.

[i] https://tial.org/publications/designing-new-institutions-ideas-and-tools-for-an-emergent-discipline/

[ii] My recent book ‘When Science Meets Power’ describes many of the potential new global institutions, from climate to technology.