A theory of knowledge atrophy

We are used to the idea that knowledge progresses, and that fields cumulate understanding over time.  This is normal in science, medicine and many other fields.  Each era knows the best knowledge of the past, and more.   Each pioneer stands on the shoulders of those who came before them.  And the more we combine different kinds of knowledge, the more knowledge as a whole accelerates forward.  Such is a dominant conventional wisdom, and it’s clearly an accurate picture in some fields. But is it true everywhere?

I think not.  I am more and more struck that in many of the fields I observe the opposite is the case.  Knowledge atrophies and decays.  Forgetting outpaces learning.  Indeed, at times it looks as if whole societies can shift from being smart to being dumb.

It’s made me conclude that the natural state of knowledge may be decay not advance.

So what distinguishes fields that cumulate knowledge from those that slip backwards and forget?  

The simple answer is that it takes hard work.   Energy has to be expended to sustain knowledge.  This is obviously true for servers and databases.  But it is also true for humans, and it takes a lot of labour for a field just to stand still, let alone to advance, even if there appear to be representations and digital stores of past knowledge.

The reasons have to do with our limited capacity to think. The stock of available knowledge is vastly greater than our capacity to understand it. So what makes the difference?  Three factors seem to be key: structure, synthesis and repetition.

The fields that advance invest heavily in ensuring solid, widely shared structures into which new thought can be added; they work hard to regularly synthesise the state of knowledge, codifying it; and they teach and summarise.  Many institutions are responsible for orchestrating collective memory, and it’s part of many people’s jobs to combat atrophy. You can see this in many areas of science  and in medicine.

But when that doesn’t happen, regression is likely.   History suggests that this is a common pattern. There is now lots of research on how civilisations can slip backwards – Tasmania’s isolation from the Australian continent is one famous example, covered well in Joseph Heinrich’s book ‘The Secrets of our Success’, which also emphasises how much we depend on social learning.  Without social reinforcements we tend to forget (and our working memory is actually weaker than other apes).

The last few thousand years have seen many similar examples.  Here in the UK, for example, the period after the collapse of the Roman Empire was a period of significant forgetting.  Peter Burke’s fascinating recent history of ignorance provides a sophisticated approach to these dynamics of learning and forgetting, although it mainly focuses on what wasn’t known in the first place rather than what was known and then forgotten. 

The topic is also engaged with by the recent interest in the multiple forms of indigenous knowledge:  essentially a concern that we have forgotten many things that could be useful to us (the book Dark Emu is a good overview of this issue in an Australian context, documenting the many sophisticated methods used for irrigation, farming and fishing prior to the arrival of the Europeans and generally forgotten).

Similar patterns are fairly well understood at an individual level in psychology.  More than a century ago Herman Ebbinghaus developed a theory of forgetting in individuals which echoes this point – he showed that memory naturally decays unless work is done to avoid that decay.  In the 20th century large bodies of research studied the links between short term and long-term memory.  But we have often lost sight of the equivalents for groups and whole societies, or perhaps assumed that they are solved by good libraries and academic disciplines, search engines and ChatGPT.

This seems to me a useful topic for new research and one relevant to the world of evidence (which is premised on the idea that cumulative knowledge can be synthesised).  It is also relevant to ‘exploratory social science’, since we surely want this too to be cumulative, to build on existing knowledge.  

It’s also relevant because the evidence on evidence use has repeatedly confirmed how hard it is to spread even the best ideas.  Social influences are more important than logic in explaining why new ideas are taken up (or more often not).  Doctors listen to other doctors, teachers to teachers, but not to others.  Ideas spread if there is an existing structure to incorporate them.  And they persist if they are integrated into training and development.

Recent research on innovations shows that even ones that achieve significant gains in performance do not naturally spread, or even survive where they first emerged.   A forthcoming study from the US shows that the key factor determining whether high impact innovations persist is whether they can be easily fitted into an existing process.  In other words, we are naturally lazy and only stick with ideas if they can be made easy – or if we are given no choice.

This echoes the findings from educational research that learning happens most easily if there is an existing ‘scaffolding’ – otherwise the more novel the idea, and the more dependent it is on complementary innovations, the more work is needed to embed the new knowledge.  I wrote about this in my book ‘Big Mind’, suggesting a framework for cognitive economics that would analyse in more detail the work needed for different kinds of intelligence and learning (from the relative easy first loop learning to the much more labour intensive second and third loop learning). 

One important field where these issues warrant attention is government. A few years ago I commissioned a study on memory in government prompted by many signs that memory was declining, despite the extensive digitisation of records which appeared at first glance to make it easier to find out about what had been done in the past.   Most observers (in the UK at least) agreed that things had got worse rather than better: corporate memory had deteriorated.  The civil service itself seemed unable to solve the problem, so instead the best recent solutions externalised the government’s memory, for example to ‘what works centres’ which gather and synthesise evidence in fields such as health and education.  Again, this mattered because policies were bound to be worse if they couldn’t draw on a body of memory of what had worked or not worked in the past.

One positive is that there are now new options for orchestrating memory and making it available.  At a minimum generative AI LLMs trained on reliable research material make it possible to provide quick syntheses of what is known (we covered these in detail recently in an event and blog which can be seen here).

The challenge though is to get these used.   Here we may want more AIs that can ask questions and act as coaches: have you thought of x?  Are you familiar with the evidence on y?  The Khan Academy’s Khanmigo is an interesting recent example, which continuously asks questions of the learner.

Reorienting LLMs towards these kinds of conversations – rather than pretending to offer answers – would be good in many ways, including in cultivating the critical thinking that is so often rare.

My simple conclusion is that a theory of knowledge atrophy and forgetting might include the following dimensions:

·        How, other things being equal, knowledge atrophies

·        How it atrophies less if there is a well curated structure for knowledge – random fragments don’t survive – which is done by professions, movements of ideas, or disciplines

·        How it atrophies less if scarce time and resources are devoted to reproducing it and repeating it, otherwise known as training and learning

·        How it atrophies less if someone has the job of systematically organising the memory – and here incentives matter: most academics are more incentivised to generate novelty than to synthesise the current state of knowledge.

I would welcome thoughts on this whole field.   Is knowledge atrophy the default for any group or organisation?  If so, how best to organise the labour to counter it?  How might we measure if a field is in decline?    And what do we do about fields that are forgetting more than they learn, since they are bound to suffer from the Dunning-Kruger effect, which means that those lacking a capability are usually unable to see that they lack it?