You already know what the Dunning-Kruger effect is

In 2017, I left academia to become a research communicator.

I thought I’d done the hard part of that job title already. I knew about research. Communication would be easy, right?

Wrong.

Very, very wrong.

Most things done with sufficient expertise look effortless to an outsider. That does not mean they are easy.

Simone Biles makes a backward double layout salto half twist out (aka the Biles I) look effortless. But unless you are an absolute twerp, it’s probably obvious to you that she’s only able to make it look effortless because she has done years and years and years of intensive practice. One does not simply walk into an Olympic stadium and Do A Big Ol' Boing.

Likewise, one does not simply walk from research into research comms and Do A Big Ol' Viral Social Media Post. It takes time and effort and practice (and, in my case at least, making it safe for academics to express their rage about their working conditions).

The good news is that there is one field in which you can achieve immediate stardom without any prior practice, and it’s called the Dunning-Kruger effect.

You already know what the Dunning-Kruger effect is, even if you’ve never heard that term before. In fact, you have seen at least one example already today.

Remember me saying that I assumed communication was the easy part of research communication? Yep, that was me giving you an example of the Dunning-Kruger effect at work: I did not know enough about a field to realise how limited my knowledge was.

It’s embarrassing to realise that you have Done A Big Ol’ Dunning-Kruger. It can be anywhere from amusing to infuriating to be on the receiving end of someone else’s ill-informed assumption that they know as much as you do about your field of expertise — it depends somewhat on who they are, the nature of their relationship with you, and how willing they are to realise that they don’t know as much as they think they do.

Sometimes, if you’re lucky, you might get to learn something when someone else hits you with a Dunning-Kruger. I don’t mean in the moment, unless you count something like a visceral insight into how Mr The Rock probably felt when Vin Diesel tried to give him acting advice. I mean later, when you’ve stopped thinking about how wrong they were, and you start thinking about how (and why) they were wrong.

I discovered this benefit entirely by accident, back when I was working at a research funder. Someone suggested I use ChatGPT to speed up my internal-facing research comms work. Feed a published journal article in, get a summary out, fact check, share with colleagues.

My first thought was, I'm 99% certain that wouldn't actually save me any time in the long run. An LLM like ChatGPT cannot understand the journal article, which means I need to fact check its summary, which means I need to read the journal article and then read the summary and check they match, and by the time I’ve done that it would have been just as quick to write the summary myself.

And then I thought, Well, okay, if it's time-neutral, maybe it's a good alternative on days when I'm not in the mood for writing?

Lol nah. Even putting aside ethical and environmental concerns, research communication is not simply summarising research. Using an LLM to summarise research would create more problems for me than it would solve. For example, it’s likely to return something focused on what the authors present as important - and I may or may not agree with them on that. (Don’t tell me you’ve never found the most interesting part of an article hidden in a footnote or a throwaway comment.)

Sure, I could write a prompt explaining what I think is important, but it would take a while, need regular updating, and would always be incomplete since some of my sense of what’s important is very difficult to put into words. Plus, I often discover what’s important about an article as I’m reading it. I can’t put information into a prompt if I don’t know it yet.

But the biggest problem with using AI to summarise research is not about the near-impossibility of writing a complete prompt.

The biggest problem is that the effort I put in would be directed away from understanding the research and towards optimising the prompt.

This might sound like a good thing, because optimising a prompt is probably less effort overall than reading and understanding and explaining a paper. By the same token, fact-checking a summary is probably less effort than creating a summary myself, so let’s examine that a bit to

I’ve done a lot of fact-checking in my life, because it is a necessary part of my work, but what I learn by doing it feels... slippery, somehow. It fades quickly. I'm left with a vague sense of I saw something about that recently rather than Ah yes, that theory came up in Oink & Crumpet's paper on hipster breakfast café names, but this new article I'm reading suggests they were wrong that cutesy names have a positive effect on people’s emotions.

Fact-checking and AI prompt-writing both require less effort than reading, understanding and explaining the research for myself. Less effort makes learning quicker, but the knowledge doesn’t stick around as long. And I need that knowledge to be able to contextualise and make sense of the things I encounter in the future.

Long-term reliance on generative AI to summarise research papers for me would, in all likelihood, slowly erode my ability to do several other parts of my job.

…and I am not sure whether I would have figured that out, at least not on a personal level, had someone with limited expertise in my area not advised me to use ChatGPT for a task that critical AI literacy indicates it’s probably not suitable for.

If you're a researcher thinking about using an LLM to summarise your own research for non-researchers, you might be thinking, None of what Clare said applies to me. I already know my research inside and out, so I can fact check at a glance. I already have the deep understanding I need to make sense of new information in my area.

True. So let's talk about a reason that applies to everyone doing research communication.

Many people assume that the core challenge of research communication is the translation element. You know, making research understandable to non-researchers without over-simplifying or introducing errors. Summarising things well.

It isn't.

The core challenge of research communication getting is your audience to trust you enough to listen to you.

The main reason my colleagues listened to me when I spoke about research was not because I knew about research. I could have written beautiful explanations and made beautiful slides and recorded beautiful videos all day long and they would not have listened to one. single. word. I said if I hadn't taken time, repeatedly, to understand what they needed from me.

So I talked with them about what their jobs were like, what they hoped research could give them, and, frequently, their worries about whether they were smart enough to make sense of it. (They were. Everyone is. If you make an honest attempt to understand something I’m explaining and you still don’t get it, that’s on me. Please tell me if that happens so that I can try to do a better job next time I’m explaining.)

Over time, my colleagues began to trust that I respected them and would do my best to understand them. Everything else came after that.

The living heart of good communication is mutual trust, respect and understanding.

An LLM cannot generate these things for you. You have to keep the heart alive for yourself, and it is messy, squishy, often extraordinarily rewarding work.

By all means consider using AI to help you create or refine words or images if you find it useful – but remember that if the heart isn't beating, the eyes will not see, the ears will not hear, and the brain will not comprehend.

If you need some help with communicating well about research, that's one of the things I specialise in. I regularly run online Practical Research Communications Skills workshops, or if you want 1:1 support you can book a research communications clinic with me.